Merge ph-fix (pluggable historian final fix wave)

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-09 14:40:45 -04:00
45 changed files with 1307 additions and 233 deletions
+4 -3
View File
@@ -1414,9 +1414,10 @@ only the recorder + outbox knobs:
hook for added historized value tags — and, since #600.1, for **changed** raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed `GatewayTagProvisioner` calls the gateway's
`EnsureTags` so a brand-new historized tag exists in the historian before the recorder's `WriteLiveValues`
land. The hook is wrapped so a faulted/synchronously-throwing provisioner can **never** block or fail a
deploy. Non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not provisioned); the
recorder likewise drops + meters non-numeric values. Continuous historization is **numeric-analog only** in
v1 (`UInt16→UInt4` is a documented fallback).
deploy. **Gateway backend:** non-numeric (`String`/`DateTime`/`Reference`) data types are skipped (not
provisioned), the recorder likewise drops + meters non-numeric values, and continuous historization is
**numeric-analog only** (`UInt16→UInt4` is a documented fallback). The **SqlServer** backend provisions and
records every type (see "Pluggable backends" above).
The provisioner is registered by the Host's `AddHistorianProvisioning` (`Runtime.ServiceCollectionExtensions`),
built via `GatewayHistorian.CreateProvisioner`, and **gated on the same `ServerHistorian:Enabled` flag as the
+4 -2
View File
@@ -225,8 +225,10 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \
Inserts, replaces, updates or deletes stored history on a historized node via the OPC UA `HistoryUpdate`
service (requires a server whose historian backend supports updates and a session holding the history-write
permission). Prints one line per item (`index time StatusCodeName`) followed by the overall service
result; any Bad item or service result exits non-zero.
permission). Prints one line per item (`index time StatusCodeName`; a code the SDK has no name for prints as
`0xXXXXXXXX`) followed by the overall service result. It exits non-zero unless **every** result is Good: a Bad
**or Uncertain** item, a non-Good node or service result, or a Good service result that is missing per-item
results (fewer than the items sent, printed as `(no result returned)`) all fail the command.
```bash
# Insert two values
+3 -2
View File
@@ -620,8 +620,9 @@ Implementation:
`GatewayAlarmHistorianWriter` (the gateway `SendEvent` path; see
[AlarmHistorian.md](AlarmHistorian.md)).
The HistorianGateway is the sole historian backend; its config keys and
deployment prerequisites are in [Historian.md](Historian.md).
The historian backend is pluggable (`ServerHistorian:Backend` = `Gateway`, the default, or `SqlServer`);
the HistorianGateway is the Gateway backend. Config keys and deployment prerequisites for both are in
[Historian.md](Historian.md).
---
+50 -12
View File
@@ -278,6 +278,10 @@ holds during the FasterLog recovery scan at start-up); the lifetime count is the
**`otopcua.historian.outbox.unreadable`** (`IHistorizationOutbox.UnreadableCount`). A growing value means
undecodable records are being skipped.
> **Metric prefixes differ.** The outbox's other two gauges are `otopcua.historization.outbox.depth` and
> `otopcua.historization.outbox.dropped`; the unreadable gauge is **`otopcua.historian.outbox.unreadable`**
> (the name the design fixed). All three describe the same outbox — search for both prefixes.
⚠️ **Rollback: drain or delete the outbox before downgrading.** A node running a pre-v2 build cannot read v2
records — its outbox deserializes every record in its constructor and fails to start on the first v2 one. Before
rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox.depth`) or stop the node and
@@ -287,8 +291,12 @@ Upgrading needs no action.
**Delivery semantics on `SqlServer`.** Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's
at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the
first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double`
cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample keeping its status** — the tag
is not quarantined for one bad reading (HistoryUpdate behaves differently: see [below](#historyupdate)). A duplicate-key
cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample** — the tag is not quarantined for
one bad reading (HistoryUpdate behaves differently: see [below](#historyupdate)). ⚠️ **An `Empty` sample is always Bad**
(the quality contract: a stored sample is never served better than "no value"): a non-finite `Double` that arrived with a
Good or Uncertain status is stored as `Empty` + **`BadOutOfRange`** (`0x803C0000`), and any other `Empty` with a non-Bad
status as `Empty` + **`BadNoData`** (`0x809B0000`). An existing Bad code — the comms-loss `BadNoCommunication` marker, a
driver's own Bad publish — is stored as it is. A duplicate-key
error (2627/2601) rolls the whole statement back, so the writer re-runs the statement (at most twice) instead of
treating the error as success.
@@ -362,21 +370,22 @@ values are counted in `RecorderStatus.SuppressedNotPrimary`, and each gate trans
## Tag auto-provisioning (EnsureTags)
When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's
`EnsureTags` **before** the first value or HistoryRead lands. `AddressSpaceApplier.Apply()` fires a
`EnsureTags` (Gateway) or the `Tag`-row `MERGE` (SqlServer) **before** the first value or HistoryRead lands. `AddressSpaceApplier.Apply()` fires a
**non-blocking, fire-and-forget** `IHistorianProvisioning.EnsureTagsAsync` for each added historized value
tag (with the resolved historian name — the `historianTagname` override when set, else the driver
`FullName`); the hook is wrapped so a faulted, slow, or throwing provisioner can **never** block or fail a
deploy. Non-numeric (`String`/`DateTime`/`Reference`) types are skipped, not provisioned.
deploy. On the **Gateway backend**, non-numeric (`String`/`DateTime`/`Reference`) types are skipped, not provisioned;
the SqlServer backend provisions every type (below).
**Wiring + gating.** The concrete provisioner comes from the selected backend's provider (`CreateProvisioning`):
the gateway-backed `GatewayTagProvisioner`, or — for `SqlServer` — `SqlServerTagProvisioner` (one `MERGE … WITH (HOLDLOCK)`
upsert per chunk of 400 tags; every data type is provisionable so `skipped` is always 0; a blank or over-400-character
name fails alone; a null unit/description in a request keeps the stored value). It is registered by the Host's
`AddHistorianProvisioning` and gated on the **same**
`ServerHistorian:Enabled` flag as the read path — both target the same single gateway. When the section is
`ServerHistorian:Enabled` flag as the read path — both use the one resolved backend. When the section is
disabled (or absent) the applier resolves the no-op `NullHistorianProvisioning` and **provisioning does not
run** (so on a read-only-historian deployment, ensure tags exist by some other means). The Host requires the
API key to carry the `historian:tags:write` scope for `EnsureTags` to succeed.
run** (so on a read-only-historian deployment, ensure tags exist by some other means). On the Gateway backend the
API key must carry the `historian:tags:write` scope for `EnsureTags` to succeed.
**Observability.** Every dispatch logs an Information tally —
`historian provisioning completed (dispatched=N, requested=…, ensured=…, skipped=…, failed=…)`. A line
@@ -530,6 +539,11 @@ paging time-based:
> `NumValuesPerNode` samples at a time across successive pages, advancing the cursor one tick past
> the timestamp once the cluster is fully drained.
>
> **Reverse reads** (`start > end`, newest first) page the same way in the opposite direction: a drained
> cluster steps the cursor one tick **down**, the read ends when that would reach the (excluded) `end`, and
> the over-fetched cluster is sliced in descending order. Before the fix a reverse read with a small
> `NumValuesPerNode` stopped after its first page with `GoodNoData`.
>
> **Short pages within a cluster still carry a continuation point.** A within-cluster page that
> returns fewer than `NumValuesPerNode` samples (because the cluster happened to be smaller than
> the cap, or is the final partial batch) is not the last page if the cluster itself has not been
@@ -643,12 +657,22 @@ everywhere (full 100 ns precision — never a `DataTable`, which would round to
transient SQL errors (connection, timeout, deadlock 1205 …), bootstrap failures, cancellation and any unclassified error;
`PermanentFail` (dead-letter) only for data errors SQL will reject forever (truncation 2628/8152, conversion, null, range, constraint).
`SourceName` is the event's `EquipmentPath`; `Severity` is the numeric Part 9 value (Low 200, Medium 500, High 700, Critical 900).
- **Tag names** are `nvarchar(400)`. `EnsureTags` fails a longer name individually, but the live-value writer's tag-create
path does **not** reject one (the parameter silently truncates) — keep historian tagnames under 400 characters.
- **Tag names** are `nvarchar(400)`, and a longer name is **refused, never truncated** (a truncating parameter would resolve
it onto whichever tag owns its first 400 characters): `EnsureTags` fails it individually, the live-value writer returns
`false` for that tag (a per-tag failure, so the #538 quarantine eventually abandons it — logged as a Warning), the updater
answers `BadInvalidArgument` for every item, and reads treat it as not found (Raw empty, AtTime / Processed `BadNoData`). An
event read filtered by a source name over 400 characters returns no events. Keep historian tagnames under 400 characters.
- **Schema bootstrap failures are cached for 5 s.** With `AutoCreateSchema=false` and the schema missing (or SQL down), every
historian call fails fast with the last bootstrap error for 5 s before SQL is tried again, so a missing schema does not
serialise every write, read and alarm event behind one round-trip each; each retry that fails logs a Warning. If
`sp_releaseapplock` fails after an auto-create, the connection pool is cleared so the session holding the applock is closed
rather than returned to the pool.
### Raw reads
`ReadRawAsync` returns every sample as stored — outage markers (`Empty` + Bad status) included.
`ReadRawAsync` returns every sample as stored — outage markers (`Empty` + Bad status) included. The one exception is defensive:
a row with no value whose stored status is **not** Bad (only a build from before the writer forced `Empty` samples Bad could
have written one) is served with `BadNoData`, on Raw and AtTime alike.
| Request | Window | Order |
|---|---|---|
@@ -674,6 +698,10 @@ outage marker and counts as Bad whatever its code). Per requested time `T` (the
| `p` usable, **`n` Bad** | **`p`'s value held** (not interpolated into the outage), `UncertainDataSubNormal` + Interpolated |
| `Double` / `Int64` with a usable `n` | linear interpolation of `NumValue`; `Int64` is computed **through `double`** and rounded to the nearest integer (ties away from zero), keeping the `Int64` type. Status `Good` + Interpolated when both neighbours are Good, else `UncertainDataSubNormal` + Interpolated |
| `Boolean` / `String` / `DateTime`, **or `T` after the last sample** | **stepped**: `p`'s value with `p`'s status + Interpolated |
| `T` after the last sample **and after now** | `p`'s value held, `UncertainDataSubNormal` + Interpolated (the value is not known to still hold) |
Setting Interpolated (or, in aggregates, Calculated) first clears any historian bit the stored status already carries, so a
result never has both.
⚠️ Because `Int64` interpolation goes through `double`, magnitudes beyond 2^53 lose precision.
@@ -791,8 +819,18 @@ mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒
`BadTypeMismatch`; and a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten. A caller-supplied status code is stored as given.
`BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a
Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status
(an operator-authored outage marker). A caller-supplied status code is otherwise stored as given.
**No DataType check.** HistoryUpdate does **not** compare a value's type with the node's `DataType`: a value is stored as sent (a `String`
written to a `Double` node's history is stored and later read back as a `String`), exactly as the live recorder stores whatever type the
driver publishes. Only a value the historian cannot represent at all is refused (`BadTypeMismatch`, above). Part 11 §6.8 permits a
server to answer `BadTypeMismatch` for a value of the wrong type; this server deliberately does not.
**`DeleteRawModifiedDetails` needs both bounds:** a `StartTime` or `EndTime` of `DateTime.MinValue` (the wire's "unspecified") is
`BadInvalidArgument` for that node and never reaches the backend.
**Duplicate timestamps inside one request** (compared at 100 ns): *Insert* — the first occurrence is applied, later ones are `BadEntryExists`;
*Replace* / *Update* — the last one wins and every earlier occurrence reports the winner's code; *DeleteAtTime* — the first reports the real
+1 -1
View File
@@ -8,7 +8,7 @@ A production OtOpcUa deployment runs **one binary per node**. The historian back
| Process | Project | Runtime | Platform | Responsibility |
|---|---|---|---|---|
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. The resolved [`EffectiveRoles`](#role-gating) decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The sole historian backend. OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The default historian backend (`ServerHistorian:Backend=Gateway`; `SqlServer` needs no sidecar). OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |
Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `docs/v2/Galaxy.ParityRig.md`); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
+3 -2
View File
@@ -2,8 +2,9 @@
> **This backend has been retired.** The bespoke Wonderware TCP/ArchestrA historian sidecar
> (`OtOpcUaWonderwareHistorian`) and its `Driver.Historian.Wonderware*` projects — plus the vestigial
> `Historian.Wonderware` driver type — were removed. **HistorianGateway is now the sole historian
> backend** for OtOpcUa (read, alarm-write, and continuous historization).
> `Historian.Wonderware` driver type — were removed. **HistorianGateway is now the default historian
> backend** for OtOpcUa (read, alarm-write, and continuous historization); a plain SQL Server backend is the
> alternative (`ServerHistorian:Backend=SqlServer`, see [Historian.md](../Historian.md)).
## What replaced it
@@ -199,8 +199,10 @@ next call (not latched).
the first value's kind, provisioning later corrects it). Insert via `{s}.SampleRow` TVP with
`INSERT … SELECT … WHERE NOT EXISTS`. Returns `true` on success, `false` on a SQL failure (the recorder's
retry/quarantine owns retry). `ServerTimeUtc = SYSUTCDATETIME()`. *As built:* duplicate timestamps within one batch
collapse to the **first**; a `NaN`/`±Inf` `Double` is stored as `Empty` keeping its status (it cannot live in a SQL
`float`); a 2627/2601 duplicate-key error rolls the statement back, so the statement is re-run (≤ 2 retries) rather
collapse to the **first**; a `NaN`/`±Inf` `Double` is stored as `Empty` (it cannot live in a SQL `float`) and, per the
final-review I-1 ruling, **an `Empty` sample is always Bad**: a non-Bad non-finite value is stored `BadOutOfRange`, any
other non-Bad `Empty` `BadNoData`, an existing Bad code kept; reads clamp a legacy non-Bad `Empty` row to `BadNoData`;
tag names over 400 characters are refused, never truncated; a 2627/2601 duplicate-key error rolls the statement back, so the statement is re-run (≤ 2 retries) rather
than treated as success.
- `IHistorianProvisioning.EnsureTagsAsync`: `MERGE` upsert of every request (all types — no numeric
filter on this backend). Per-request failures are counted, never thrown.
@@ -273,7 +275,8 @@ public interface IHistorianDataUpdater
Each call is one transaction (`MERGE … OUTPUT $action` against the TVP). A value with a null timestamp ⇒
`BadInvalidTimestamp` for that item. *As built:* a `NaN`/`±Inf` `Double` ⇒ `BadInvalidArgument` (rejected, not stored as `Empty`
as the live-value writer does); duplicate timestamps in one request — Insert: first applies, later `BadEntryExists`; Replace/Update:
as the live-value writer does), and so is a null value with a non-Bad status (I-1); no DataType check against the node (M-1,
decided: values are stored as sent); a `DeleteRawModified` with a `MinValue` bound ⇒ `BadInvalidArgument`; duplicate timestamps in one request — Insert: first applies, later `BadEntryExists`; Replace/Update:
last wins, earlier ones share its code; DeleteAtTime: first reports, later `BadNoEntryExists`. The node manager answers a null item
`BadInvalidArgument`, `MinValue` `BadInvalidTimestamp`, an untypable value `BadTypeMismatch`, a backend throw `BadInternalError`
and a deadline `BadTimeout` (a failure is not "unsupported"). Update/Insert on an unknown tag creates the tag row (same as live
@@ -159,32 +159,54 @@ public class HistoryUpdateCommand : CommandBase
await console.Output.WriteLineAsync($"HistoryUpdate {Mode.Trim().ToLowerInvariant()} on {NodeId}");
await console.Output.WriteLineAsync($"{"Index",-6} {"Time",-35} {"Status"}");
var anyBad = false;
// Exit non-zero unless EVERY expected result came back Good: a Bad or Uncertain item, a Bad result or
// service status, or a Good service result that is missing results (the server answered fewer items
// than were sent) all fail the command.
var failed = false;
var expected = ExpectedOperationCount(details);
var ops = outcome.OperationResults.Count > 0 ? outcome.OperationResults[0] : [];
if (ops.Count > 0)
if (expected > 0)
{
for (var i = 0; i < ops.Count; i++)
for (var i = 0; i < Math.Max(expected, ops.Count); i++)
{
var time = i < times.Count ? times[i] : "-";
anyBad |= StatusCode.IsBad(ops[i]);
await console.Output.WriteLineAsync($"{i,-6} {time,-35} {StatusCodes.GetBrowseName(ops[i].Code)}");
string status;
if (i < ops.Count)
{
failed |= !StatusCode.IsGood(ops[i]);
status = StatusName(ops[i]);
}
else
{
failed = true;
status = "(no result returned)";
}
await console.Output.WriteLineAsync($"{i,-6} {time,-35} {status}");
}
}
else if (outcome.ResultStatus.Count > 0)
{
// delete-raw reports a single result-level status for the whole range.
anyBad |= StatusCode.IsBad(outcome.ResultStatus[0]);
await console.Output.WriteLineAsync(
$"{0,-6} {(times.Count > 0 ? times[0] : "-"),-35} {StatusCodes.GetBrowseName(outcome.ResultStatus[0].Code)}");
$"{0,-6} {(times.Count > 0 ? times[0] : "-"),-35} {StatusName(outcome.ResultStatus[0])}");
}
var overall = StatusCodes.GetBrowseName(outcome.ServiceResult.Code);
await console.Output.WriteLineAsync($"Service result: {overall}");
anyBad |= StatusCode.IsBad(outcome.ServiceResult);
if (outcome.ResultStatus.Count > 0 && StatusCode.IsBad(outcome.ResultStatus[0])) anyBad = true;
if (outcome.ResultStatus.Count == 0)
{
failed = true;
await console.Output.WriteLineAsync("No history-update result was returned for the node.");
}
else
{
failed |= !StatusCode.IsGood(outcome.ResultStatus[0]);
}
if (anyBad)
throw new CommandException("HistoryUpdate reported one or more Bad results.");
await console.Output.WriteLineAsync($"Service result: {StatusName(outcome.ServiceResult)}");
failed |= !StatusCode.IsGood(outcome.ServiceResult);
if (failed)
throw new CommandException(
"HistoryUpdate did not report Good for every item (a Bad or Uncertain result, or a missing result).");
}
finally
{
@@ -196,6 +218,27 @@ public class HistoryUpdateCommand : CommandBase
}
}
/// <summary>How many per-item operation results the server owes for <paramref name="details"/> (0 for delete-raw).</summary>
private static int ExpectedOperationCount(HistoryUpdateDetails details) => details switch
{
UpdateDataDetails u => u.UpdateValues.Count,
DeleteAtTimeDetails d => d.ReqTimes.Count,
_ => 0
};
/// <summary>
/// The SDK's symbolic name for <paramref name="code"/>, or its hex value when the SDK has none — a status is
/// never printed blank.
/// </summary>
/// <param name="code">The status code to describe.</param>
/// <returns>The status name, or <c>0xXXXXXXXX</c>.</returns>
private static string StatusName(StatusCode code)
{
var name = StatusCodes.GetBrowseName(code.Code);
if (string.IsNullOrEmpty(name)) name = StatusCodes.GetBrowseName(code.Code & 0xFFFF0000u);
return string.IsNullOrEmpty(name) ? $"0x{code.Code:X8}" : name;
}
private static List<string> DescribeTimes(HistoryUpdateDetails details)
{
return details switch
@@ -1,13 +1,13 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions;
/// <summary>
/// One numeric sample the continuous-historization recorder drains to the historian's SQL
/// live-value write path: an optional UTC timestamp, the typed value, and the OPC UA status code
/// the sample was recorded with.
/// One sample the continuous-historization recorder drains to the backend's live-value write path: an
/// optional UTC timestamp, the typed value (any <see cref="HistorianValueKind"/> the backend supports — the
/// Gateway backend takes numeric kinds only, SqlServer takes all of them), and the OPC UA status code the
/// sample was recorded with.
/// </summary>
/// <param name="TimestampUtc">
/// UTC source timestamp of the sample, or <c>null</c> to defer to the historian's server-stamped
/// current-time path (the SQL writer uses <c>SYSDATETIME()</c> when the timestamp is absent).
/// UTC source timestamp of the sample, or <c>null</c> to have the backend stamp the current time.
/// </param>
/// <param name="Value">The typed sample value.</param>
/// <param name="StatusCode">OPC UA status code of the sample (0 = Good).</param>
@@ -16,20 +16,20 @@ public readonly record struct HistorizationValue(DateTime? TimestampUtc, Histori
/// <summary>
/// Seam over the historian's live-value write path used by the continuous-historization recorder.
/// Lives in the abstraction layer so the Runtime recorder depends on it without taking a hard
/// reference on the gRPC gateway driver; the gateway driver supplies the concrete adapter
/// (<c>GatewayHistorianValueWriter</c>).
/// reference on a backend driver; the selected backend provider supplies the concrete writer
/// (<c>GatewayHistorianValueWriter</c> or <c>SqlServerHistorianValueWriter</c>).
/// </summary>
public interface IHistorianValueWriter
{
/// <summary>
/// Writes a batch of live values for a single tag through the historian's SQL live-write path.
/// Writes a batch of live values for a single tag through the backend's live-write path.
/// Implementations are expected to be non-throwing: a transport/gateway error is surfaced as a
/// <c>false</c> result so the recorder retains the entries and retries, rather than as an
/// exception.
/// </summary>
/// <param name="tag">Fully-qualified historian tag the values are recorded against.</param>
/// <param name="values">The numeric samples to write, in append order.</param>
/// <param name="values">The samples to write, in append order.</param>
/// <param name="ct">Cancellation token.</param>
/// <returns><c>true</c> on a successful (or durably-queued) gateway ack; <c>false</c> on a retryable failure.</returns>
/// <returns><c>true</c> when the backend stored (or durably queued) the batch; <c>false</c> on a retryable failure.</returns>
Task<bool> WriteLiveValuesAsync(string tag, IReadOnlyList<HistorizationValue> values, CancellationToken ct);
}
@@ -14,13 +14,6 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// </summary>
public sealed class GatewayHistorianProvider : IHistorianBackendProvider
{
private static readonly IReadOnlySet<HistorianValueKind> Kinds = new HashSet<HistorianValueKind>
{
HistorianValueKind.Double,
HistorianValueKind.Int64,
HistorianValueKind.Boolean,
};
private readonly ServerHistorianOptions _options;
/// <summary>Creates the provider over the bound <c>ServerHistorian</c> options.</summary>
@@ -34,7 +27,11 @@ public sealed class GatewayHistorianProvider : IHistorianBackendProvider
public string Name => "Gateway";
/// <inheritdoc />
public IReadOnlySet<HistorianValueKind> SupportedKinds => Kinds;
/// <remarks>
/// The recorder's own default set (<see cref="ContinuousHistorizationRecorder.DefaultSupportedKinds"/>), so the
/// gateway's kinds and the recorder's fallback can never drift apart.
/// </remarks>
public IReadOnlySet<HistorianValueKind> SupportedKinds => ContinuousHistorizationRecorder.DefaultSupportedKinds;
/// <inheritdoc />
public IHistorianDataSource CreateDataSource(IServiceProvider sp) => GatewayHistorian.CreateDataSource(_options, sp);
@@ -28,12 +28,18 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Recorder;
/// <b>Quality is a documented-dead input</b> on the <c>WriteLiveValues</c> SQL path
/// (HistorianGateway 0.2.0 Contracts C-002): <c>HistorianLiveValue.Quality</c> /
/// <c>QualityDetail</c> are server-managed columns — any value sent is accepted on the wire and
/// then silently discarded. We populate <c>Quality</c> anyway (see the mapping below): proto3
/// transmits <c>0</c> when a field is left unset, and <c>0</c> is "Bad" in OPC-DA terms, so
/// <em>not</em> setting it would send a <em>wrong</em> quality to any future gateway version that
/// starts honoring the field — whereas the recorder's <c>192</c> ("Good") is the correct value.
/// <b>Do NOT extend the recorder to capture real node quality expecting it to persist</b> — it
/// will not, until the gateway honors the field.
/// then silently discarded. We populate <c>Quality</c> anyway: proto3 transmits <c>0</c> when a
/// field is left unset, and <c>0</c> is "Bad" in OPC-DA terms, so <em>not</em> setting it would send a
/// <em>wrong</em> quality to any future gateway version that starts honoring the field. The recorder
/// now carries each sample's real OPC UA status, which <see cref="Mapping.GatewayQualityMapper.ToDaQuality"/>
/// maps to the DA byte (Good ⇒ <c>192</c>, as before). <b>That status does not persist on this
/// backend</b>: the gateway discards the field, so AVEVA still stores every sample Good — use the
/// SqlServer backend where the stored status must be the real one.
/// </para>
/// <para>
/// <b>Numeric only.</b> Values whose kind has no double form (String, DateTime, Empty) are skipped,
/// never thrown; a batch with none left is acked without calling the gateway. The recorder already drops
/// those kinds at capture for this backend, so the skip is a backstop.
/// </para>
/// </remarks>
public sealed class GatewayHistorianValueWriter : IHistorianValueWriter, IAsyncDisposable
@@ -0,0 +1,27 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// The one place that maps value kinds to <c>Tag.DataType</c> (the schema stores a <see cref="DriverDataType"/> byte).
/// Used by the value writer and the updater only when they have to create a tag row on a miss; provisioning
/// later corrects it.
/// </summary>
internal static class SqlHistorianDataTypes
{
/// <summary>Data type used when a tag is created from a batch that holds no typed value.</summary>
public const DriverDataType DefaultForEmpty = DriverDataType.Float64;
/// <summary>Maps a value kind to the stored <c>Tag.DataType</c> byte (Empty ⇒ <see cref="DefaultForEmpty"/>).</summary>
/// <param name="kind">Value kind.</param>
/// <returns>The <see cref="DriverDataType"/> as a byte.</returns>
public static byte FromKind(HistorianValueKind kind) => (byte)(kind switch
{
HistorianValueKind.Double => DriverDataType.Float64,
HistorianValueKind.Int64 => DriverDataType.Int64,
HistorianValueKind.Boolean => DriverDataType.Boolean,
HistorianValueKind.String => DriverDataType.String,
HistorianValueKind.DateTime => DriverDataType.DateTime,
_ => DefaultForEmpty,
});
}
@@ -7,17 +7,46 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
internal static class SqlHistorianStatusCodes
{
public const uint Good = 0x00000000u;
public const uint GoodNoData = 0x00A50000u;
public const uint GoodEntryInserted = 0x00A20000u;
public const uint GoodEntryReplaced = 0x00A30000u;
public const uint BadNoData = 0x809B0000u;
public const uint BadEntryExists = 0x809F0000u;
public const uint BadNoEntryExists = 0x80A00000u;
public const uint BadInvalidTimestamp = 0x80230000u;
public const uint BadTypeMismatch = 0x80740000u;
public const uint BadAggregateInvalidInputs = 0x80D60000u;
public const uint BadHistoryOperationUnsupported = 0x80720000u;
public const uint UncertainDataSubNormal = 0x40A40000u;
/// <summary>Stored for a non-finite Double (NaN / ±Infinity) that arrived with a non-Bad status.</summary>
public const uint BadOutOfRange = 0x803C0000u;
}
/// <summary>
/// Bit masks over a 32-bit OPC UA status code (Part 4 §7.39). These are masks, not emitted codes, so their
/// names deliberately avoid the Good/Bad/Uncertain prefix the status-code parity scan keys on.
/// </summary>
internal static class StatusBits
{
/// <summary>Severity + sub-code (top 16 bits) of a status, i.e. the code without info bits.</summary>
public const uint CodeBitsMask = 0xFFFF0000u;
/// <summary>The two severity bits (Good = 00, Uncertain = 01, Bad = 10).</summary>
public const uint SeverityMask = 0xC0000000u;
/// <summary>The severity bit set on every Bad code.</summary>
public const uint SeverityBadBit = 0x80000000u;
/// <summary>The two historian info bits (Calculated, Interpolated) in the low bits of the info field.</summary>
public const uint HistorianBitsMask = 0x3u;
/// <summary>True when <paramref name="status"/> has Bad severity.</summary>
/// <param name="status">A status code.</param>
/// <returns>Whether the Bad severity bit is set.</returns>
public static bool IsBad(uint status) => (status & SeverityBadBit) != 0;
/// <summary>True when <paramref name="status"/> has Good severity.</summary>
/// <param name="status">A status code.</param>
/// <returns>Whether both severity bits are clear.</returns>
public static bool IsGood(uint status) => (status & SeverityMask) == 0;
}
/// <summary>Part 4 §7.39 historian info bits OR-ed onto a status code.</summary>
@@ -32,13 +61,21 @@ internal static class HistorianInfoBits
/// <summary>The value was interpolated.</summary>
public const uint HistorianInterpolated = 0x2u;
/// <summary>Marks <paramref name="status"/> as a Calculated data value.</summary>
/// <summary>
/// Marks <paramref name="status"/> as a Calculated data value. Any historian bits the status already carries
/// (e.g. a stored Interpolated sample) are cleared first, so the result is never both.
/// </summary>
/// <param name="status">Base status code.</param>
/// <returns>Status with InfoType and Calculated bits set.</returns>
public static uint WithCalculated(uint status) => status | InfoTypeDataValue | HistorianCalculated;
public static uint WithCalculated(uint status) =>
(status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianCalculated;
/// <summary>Marks <paramref name="status"/> as an Interpolated data value.</summary>
/// <summary>
/// Marks <paramref name="status"/> as an Interpolated data value. Any historian bits the status already
/// carries (e.g. a stored Calculated sample) are cleared first, so the result is never both.
/// </summary>
/// <param name="status">Base status code.</param>
/// <returns>Status with InfoType and Interpolated bits set.</returns>
public static uint WithInterpolated(uint status) => status | InfoTypeDataValue | HistorianInterpolated;
public static uint WithInterpolated(uint status) =>
(status & ~StatusBits.HistorianBitsMask) | InfoTypeDataValue | HistorianInterpolated;
}
@@ -95,7 +95,7 @@ internal sealed class SqlServerAlarmHistorianWriter : IAlarmHistorianWriter
{
return HistorianWriteOutcome.Ack;
}
}
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return HistorianWriteOutcome.RetryPlease;
@@ -34,6 +34,14 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// markers included. Aggregates exclude Bad samples from the math, report the interval Uncertain when
/// any non-Good sample is present, and count them in PercentBad. AtTime never interpolates across a Bad
/// neighbour and reports a time inside an outage with the outage's Bad code (see <see cref="ReadAtTimeAsync"/>).
/// <b>An Empty sample is always served Bad:</b> a row with no value whose stored status is not Bad (only
/// possible for rows written before the writer forced Empty samples Bad) is read back as <c>BadNoData</c>,
/// on Raw and AtTime alike.
/// </para>
/// <para>
/// <b>Long names.</b> A tag name or event source name longer than <see cref="SqlServerTagCatalog.MaxNameLength"/>
/// can never be stored, so it is answered as not found (Raw: empty; AtTime / Processed: <c>BadNoData</c>;
/// Events: none) without being truncated onto another series.
/// </para>
/// </remarks>
public sealed class SqlServerHistorianDataSource : IHistorianDataSource
@@ -41,15 +49,13 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// <summary>Spec §9.7: a processed read covering more intervals than this is refused.</summary>
public const int MaxProcessedBuckets = 100_000;
/// <summary>Severity + sub-code (top 16 bits) of a status, i.e. the code without info bits.</summary>
private const uint StatusCodeBitsMask = 0xFFFF0000u;
private const string SampleColumns = "SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc";
private readonly SqlHistorianConnection _conn;
private readonly SqlServerSchemaBootstrapper _bootstrap;
private readonly SqlServerTagCatalog _catalog;
private readonly ILogger _log;
private readonly TimeProvider _time;
private readonly object _healthLock = new();
private DateTime? _lastSuccessUtc;
@@ -65,13 +71,16 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// <param name="bootstrap">Schema bootstrapper (shared with the other SQL seams).</param>
/// <param name="catalog">Tag catalog (shared; reads use only its read-only lookup).</param>
/// <param name="log">Optional logger.</param>
/// <param name="timeProvider">Clock for "now" (AtTime beyond the last sample, server timestamps); system by default.</param>
internal SqlServerHistorianDataSource(
SqlHistorianConnection conn, SqlServerSchemaBootstrapper bootstrap, SqlServerTagCatalog catalog, ILogger? log = null)
SqlHistorianConnection conn, SqlServerSchemaBootstrapper bootstrap, SqlServerTagCatalog catalog, ILogger? log = null,
TimeProvider? timeProvider = null)
{
_conn = conn ?? throw new ArgumentNullException(nameof(conn));
_bootstrap = bootstrap ?? throw new ArgumentNullException(nameof(bootstrap));
_catalog = catalog ?? throw new ArgumentNullException(nameof(catalog));
_log = log ?? NullLogger.Instance;
_time = timeProvider ?? TimeProvider.System;
}
// ------------------------------------------------------------------------------------------ Raw
@@ -149,7 +158,8 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// neighbours are Good, else <c>UncertainDataSubNormal</c> + Interpolated (never better than the
/// inputs).</item>
/// <item>Boolean/String/DateTime, or no <c>n</c> (after the last sample) ⇒ stepped: <c>p</c>'s value with
/// <c>p</c>'s status code + Interpolated bits.</item>
/// <c>p</c>'s status code + Interpolated bits — except that a time later than now that is stepped from the
/// last sample is <c>UncertainDataSubNormal</c> + Interpolated (the value is not known to still hold).</item>
/// </list>
/// Unknown tag ⇒ every entry <c>BadNoData</c>.
/// </remarks>
@@ -160,7 +170,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
ArgumentNullException.ThrowIfNull(timestampsUtc);
try
{
var now = DateTime.UtcNow;
var now = _time.GetUtcNow().UtcDateTime;
var byTicks = new Dictionary<long, DataValueSnapshot>(timestampsUtc.Count);
var tagId = timestampsUtc.Count == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
if (tagId is { } id)
@@ -182,7 +192,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
StoredSample? p = reader.IsDBNull(1) ? null : ReadStored(reader, 1);
StoredSample? n = reader.IsDBNull(8) ? null : ReadStored(reader, 8);
var serverTime = p is { } ps && ps.TimeUtc == t ? SqlHistorianConnection.GetUtc(reader, 7) : now;
byTicks[t.Ticks] = AtTimeValue(t, p, n, serverTime);
byTicks[t.Ticks] = AtTimeValue(t, p, n, serverTime, now);
}
}
@@ -206,7 +216,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
}
/// <summary>Applies the AtTime rules (see <see cref="ReadAtTimeAsync"/>) to one requested time.</summary>
private static DataValueSnapshot AtTimeValue(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime)
private static DataValueSnapshot AtTimeValue(DateTime t, StoredSample? prev, StoredSample? next, DateTime serverTime, DateTime now)
{
if (prev is not { } p)
return new DataValueSnapshot(null, SqlHistorianStatusCodes.BadNoData, t, serverTime);
@@ -215,7 +225,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
// Inside an outage: report the outage's own Bad code, never an upgraded Uncertain.
if (!p.IsUsable)
return new DataValueSnapshot(null, p.Status & StatusCodeBitsMask, t, serverTime);
return new DataValueSnapshot(null, p.Status & StatusBits.CodeBitsMask, t, serverTime);
// Usable previous, Bad next: hold the previous value up to the outage; do not interpolate into it.
if (next is { IsUsable: false })
@@ -234,6 +244,11 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
return new DataValueSnapshot(value, HistorianInfoBits.WithInterpolated(status), t, serverTime);
}
// Beyond the last sample AND beyond now: the value is held into the future, so it is not known to be Good.
if (next is null && t > now)
return new DataValueSnapshot(p.Value.ToObject(),
HistorianInfoBits.WithInterpolated(SqlHistorianStatusCodes.UncertainDataSubNormal), t, serverTime);
// Stepped (non-numeric kind, or after the last sample): hold the previous value and its status.
return new DataValueSnapshot(p.Value.ToObject(), HistorianInfoBits.WithInterpolated(p.Status), t, serverTime);
}
@@ -252,6 +267,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken)
{
// No stored event can carry a source longer than the column (the alarm writer dead-letters it rather than
// truncating), so an over-long filter matches nothing — never a truncated prefix of someone else's source.
if (sourceName is not null && sourceName.Length > SqlServerTagCatalog.MaxNameLength)
return new HistoricalEventsResult([], null);
try
{
await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false);
@@ -269,7 +288,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
SqlHistorianConnection.AddDateTime2(cmd, "@lo", lo);
SqlHistorianConnection.AddDateTime2(cmd, "@hi", hi);
if (sourceName is not null)
cmd.Parameters.Add(new SqlParameter("@source", SqlDbType.NVarChar, 400) { Value = sourceName });
cmd.Parameters.Add(new SqlParameter("@source", SqlDbType.NVarChar, SqlServerTagCatalog.MaxNameLength) { Value = sourceName });
var events = new List<HistoricalEvent>();
byte[]? continuation = null;
@@ -349,7 +368,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
try
{
var now = DateTime.UtcNow;
var now = _time.GetUtcNow().UtcDateTime;
var rows = new Dictionary<long, BucketRow>();
var tagId = bucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
if (tagId is { } id)
@@ -492,21 +511,29 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
private readonly record struct StoredSample(DateTime TimeUtc, HistorianValue Value, uint Status)
{
/// <summary>Severity Good or Uncertain, and a value present (an Empty sample is an outage marker).</summary>
public bool IsUsable => (Status & 0x80000000u) == 0 && Value.Kind != HistorianValueKind.Empty;
public bool IsUsable => !StatusBits.IsBad(Status) && Value.Kind != HistorianValueKind.Empty;
/// <summary>Usable with severity Good.</summary>
public bool IsGood => IsUsable && (Status & 0xC0000000u) == 0;
public bool IsGood => IsUsable && StatusBits.IsGood(Status);
}
/// <summary>Reads <c>SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode</c> starting at <paramref name="o"/>.</summary>
private static StoredSample ReadStored(SqlDataReader r, int o) => new(
SqlHistorianConnection.GetUtc(r, o),
SampleRowMapper.FromColumns(
/// <summary>
/// Reads <c>SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode</c> starting at <paramref name="o"/>.
/// An Empty (no-value) row whose stored status is not Bad is clamped to <c>BadNoData</c> — defence in depth for
/// rows written before the writer forced every Empty sample Bad.
/// </summary>
private static StoredSample ReadStored(SqlDataReader r, int o)
{
var value = SampleRowMapper.FromColumns(
r.GetByte(o + 1),
r.IsDBNull(o + 2) ? null : r.GetDouble(o + 2),
r.IsDBNull(o + 3) ? null : r.GetInt64(o + 3),
r.IsDBNull(o + 4) ? null : r.GetString(o + 4)),
SampleRowMapper.FromDbStatus(r.GetInt32(o + 5)));
r.IsDBNull(o + 4) ? null : r.GetString(o + 4));
var status = SampleRowMapper.FromDbStatus(r.GetInt32(o + 5));
if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status))
status = SqlHistorianStatusCodes.BadNoData;
return new StoredSample(SqlHistorianConnection.GetUtc(r, o), value, status);
}
/// <summary>Read-only tag lookup (never creates a row); runs the schema bootstrap on a cache miss.</summary>
private Task<int?> ResolveAsync(string name, CancellationToken ct) => _catalog.TryGetTagIdAsync(name, ct);
@@ -6,7 +6,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>Status codes owned by the updater (named consts so <c>StatusCodeParityTests</c> sees them).</summary>
internal static class SqlHistorianUpdateStatusCodes
{
/// <summary>A value that cannot be stored (Double NaN / ±Infinity).</summary>
/// <summary>
/// A value that cannot be stored: a Double NaN / ±Infinity, an Empty (null) value with a non-Bad status, or a
/// tag name longer than <see cref="SqlServerTagCatalog.MaxNameLength"/>.
/// </summary>
public const uint BadInvalidArgument = 0x80AB0000u;
}
@@ -19,7 +22,13 @@ internal static class SqlHistorianUpdateStatusCodes
/// DeleteAtTime — the first occurrence reports the real outcome, later ones <c>BadNoEntryExists</c>.</para>
/// <para><b>Invalid items.</b> A null or <see cref="DateTime.MinValue"/> timestamp is
/// <c>BadInvalidTimestamp</c>. A Double NaN/±Infinity cannot live in the SQL <c>float</c> column and is
/// rejected with <c>BadInvalidArgument</c> (not silently stored as Empty). Other items still proceed.</para>
/// rejected with <c>BadInvalidArgument</c> (not silently stored as Empty). An Empty (null) value is only
/// accepted with a Bad status (an operator-authored outage marker); with a Good or Uncertain status it is
/// <c>BadInvalidArgument</c>, because a stored sample is never served better than "no value". Other items
/// still proceed.</para>
/// <para><b>Tag names over <see cref="SqlServerTagCatalog.MaxNameLength"/> characters</b> are refused, never
/// truncated: every item of the call (and DeleteRaw's single result) reports <c>BadInvalidArgument</c> and nothing
/// is written.</para>
/// <para><b>Unknown tag.</b> Insert/Update create it via <see cref="SqlServerTagCatalog"/>; Replace and the
/// deletes report <c>BadNoEntryExists</c> (DeleteRaw: <c>BadNoData</c>) without creating anything.</para>
/// </summary>
@@ -44,6 +53,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(values);
var results = new uint[values.Count];
if (!SqlServerTagCatalog.IsValidName(tag))
{
Array.Fill(results, SqlHistorianUpdateStatusCodes.BadInvalidArgument);
return results;
}
var winner = new Dictionary<long, int>(); // tick -> index of the item actually sent
for (var i = 0; i < values.Count; i++)
{
@@ -53,7 +67,8 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
results[i] = SqlHistorianStatusCodes.BadInvalidTimestamp;
continue;
}
if (v.Value.Kind == HistorianValueKind.Double && !double.IsFinite(v.Value.Number))
if ((v.Value.Kind == HistorianValueKind.Double && !double.IsFinite(v.Value.Number))
|| (v.Value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(v.StatusCode)))
{
results[i] = SqlHistorianUpdateStatusCodes.BadInvalidArgument;
continue;
@@ -71,7 +86,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
var tagId = mode == HistoryUpdateMode.Replace
? await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false)
: await _tags.GetOrCreateTagIdAsync(tag, DataTypeFor(values[send[0]].Value.Kind), ct).ConfigureAwait(false);
: await _tags.GetOrCreateTagIdAsync(tag, SqlHistorianDataTypes.FromKind(values[send[0]].Value.Kind), ct).ConfigureAwait(false);
var outcome = tagId is null
? new Dictionary<long, uint>()
@@ -98,6 +113,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
public async Task<uint> DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct)
{
ArgumentNullException.ThrowIfNull(tag);
if (!SqlServerTagCatalog.IsValidName(tag)) return SqlHistorianUpdateStatusCodes.BadInvalidArgument;
var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false);
if (tagId is null) return SqlHistorianStatusCodes.BadNoData;
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
@@ -121,6 +137,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(timesUtc);
var results = new uint[timesUtc.Count];
if (!SqlServerTagCatalog.IsValidName(tag))
{
Array.Fill(results, SqlHistorianUpdateStatusCodes.BadInvalidArgument);
return results;
}
var first = new Dictionary<long, int>();
for (var i = 0; i < timesUtc.Count; i++)
{
@@ -177,14 +198,4 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
await tx.CommitAsync(ct).ConfigureAwait(false);
return outcome;
}
// T9 owns the canonical HistorianValueKind -> Tag.DataType mapping; a later fix wave will dedupe the two.
private static byte DataTypeFor(HistorianValueKind kind) => (byte)(kind switch
{
HistorianValueKind.Int64 => DriverDataType.Int64,
HistorianValueKind.Boolean => DriverDataType.Boolean,
HistorianValueKind.String => DriverDataType.String,
HistorianValueKind.DateTime => DriverDataType.DateTime,
_ => DriverDataType.Float64,
});
}
@@ -1,3 +1,4 @@
using System.Collections.Frozen;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
@@ -11,23 +12,26 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <see cref="HistorianValueKind"/> and HistoryUpdate. All seams share ONE connection helper, ONE schema
/// bootstrapper and ONE tag catalog, so the schema is ensured once and tag ids are cached once.
/// </summary>
/// <remarks>
/// The shared state is built on the first <c>Create*</c> call, not in the constructor: the constructor does no
/// validation and no I/O, so resolving the provider for a backend nobody consumes never fails on its options
/// (they are validated, consumer-gated, by the host's <c>ServerHistorianOptionsValidator</c>). The first
/// <c>Create*</c> call's service provider supplies the logger factory the bootstrapper logs through.
/// </remarks>
public sealed class SqlServerHistorianProvider : IHistorianBackendProvider
{
private static readonly IReadOnlySet<HistorianValueKind> Kinds =
new HashSet<HistorianValueKind>(Enum.GetValues<HistorianValueKind>());
private static readonly FrozenSet<HistorianValueKind> Kinds =
Enum.GetValues<HistorianValueKind>().ToFrozenSet();
private readonly SqlHistorianConnection _conn;
private readonly SqlServerSchemaBootstrapper _bootstrap;
private readonly SqlServerTagCatalog _catalog;
private readonly SqlServerHistorianOptions _options;
private readonly object _initLock = new();
private Shared? _shared;
/// <summary>Creates the provider over validated options.</summary>
/// <summary>Creates the provider over the <c>ServerHistorian:SqlServer</c> options (validated elsewhere).</summary>
/// <param name="options">The <c>ServerHistorian:SqlServer</c> options.</param>
public SqlServerHistorianProvider(SqlServerHistorianOptions options)
{
ArgumentNullException.ThrowIfNull(options);
_conn = new SqlHistorianConnection(options);
_bootstrap = new SqlServerSchemaBootstrapper(_conn, options);
_catalog = new SqlServerTagCatalog(_conn, _bootstrap);
_options = options ?? throw new ArgumentNullException(nameof(options));
}
/// <inheritdoc />
@@ -36,26 +40,61 @@ public sealed class SqlServerHistorianProvider : IHistorianBackendProvider
/// <inheritdoc />
public IReadOnlySet<HistorianValueKind> SupportedKinds => Kinds;
private static ILogger Log<T>(IServiceProvider sp) =>
((sp.GetService(typeof(ILoggerFactory)) as ILoggerFactory) ?? NullLoggerFactory.Instance).CreateLogger<T>();
private static ILoggerFactory Loggers(IServiceProvider sp) =>
(sp.GetService(typeof(ILoggerFactory)) as ILoggerFactory) ?? NullLoggerFactory.Instance;
private Shared Get(IServiceProvider sp)
{
if (Volatile.Read(ref _shared) is { } built) return built;
lock (_initLock)
{
if (_shared is { } raced) return raced;
var conn = new SqlHistorianConnection(_options);
var bootstrap = new SqlServerSchemaBootstrapper(conn, _options,
Loggers(sp).CreateLogger<SqlServerSchemaBootstrapper>());
var shared = new Shared(conn, bootstrap, new SqlServerTagCatalog(conn, bootstrap));
Volatile.Write(ref _shared, shared);
return shared;
}
}
/// <inheritdoc />
public IHistorianDataSource CreateDataSource(IServiceProvider sp) =>
new SqlServerHistorianDataSource(_conn, _bootstrap, _catalog, Log<SqlServerHistorianDataSource>(sp));
public IHistorianDataSource CreateDataSource(IServiceProvider sp)
{
var s = Get(sp);
return new SqlServerHistorianDataSource(s.Conn, s.Bootstrap, s.Catalog,
Loggers(sp).CreateLogger<SqlServerHistorianDataSource>());
}
/// <inheritdoc />
public IHistorianValueWriter CreateValueWriter(IServiceProvider sp) =>
new SqlServerHistorianValueWriter(_conn, _bootstrap, _catalog, Log<SqlServerHistorianValueWriter>(sp));
public IHistorianValueWriter CreateValueWriter(IServiceProvider sp)
{
var s = Get(sp);
return new SqlServerHistorianValueWriter(s.Conn, s.Bootstrap, s.Catalog,
Loggers(sp).CreateLogger<SqlServerHistorianValueWriter>());
}
/// <inheritdoc />
public IAlarmHistorianWriter CreateAlarmWriter(IServiceProvider sp) =>
new SqlServerAlarmHistorianWriter(_conn, _bootstrap, Log<SqlServerAlarmHistorianWriter>(sp));
public IAlarmHistorianWriter CreateAlarmWriter(IServiceProvider sp)
{
var s = Get(sp);
return new SqlServerAlarmHistorianWriter(s.Conn, s.Bootstrap,
Loggers(sp).CreateLogger<SqlServerAlarmHistorianWriter>());
}
/// <inheritdoc />
public IHistorianProvisioning CreateProvisioning(IServiceProvider sp) =>
new SqlServerTagProvisioner(_conn, _bootstrap, Log<SqlServerTagProvisioner>(sp));
public IHistorianProvisioning CreateProvisioning(IServiceProvider sp)
{
var s = Get(sp);
return new SqlServerTagProvisioner(s.Conn, s.Bootstrap, Loggers(sp).CreateLogger<SqlServerTagProvisioner>());
}
/// <inheritdoc />
public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp) =>
new SqlServerHistorianDataUpdater(_conn, _catalog);
public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp)
{
var s = Get(sp);
return new SqlServerHistorianDataUpdater(s.Conn, s.Catalog);
}
private sealed record Shared(SqlHistorianConnection Conn, SqlServerSchemaBootstrapper Bootstrap, SqlServerTagCatalog Catalog);
}
@@ -5,29 +5,6 @@ using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// The one place that maps value kinds to <c>Tag.DataType</c> (the schema stores a <see cref="DriverDataType"/> byte).
/// Used only when the writer has to create a tag row on a miss; provisioning later corrects it.
/// </summary>
internal static class SqlHistorianDataTypes
{
/// <summary>Data type used when a tag is created from a batch that holds no typed value.</summary>
public const DriverDataType DefaultForEmpty = DriverDataType.Float64;
/// <summary>Maps a value kind to the stored <c>Tag.DataType</c> byte (Empty ⇒ <see cref="DefaultForEmpty"/>).</summary>
/// <param name="kind">Value kind.</param>
/// <returns>The <see cref="DriverDataType"/> as a byte.</returns>
public static byte FromKind(HistorianValueKind kind) => (byte)(kind switch
{
HistorianValueKind.Double => DriverDataType.Float64,
HistorianValueKind.Int64 => DriverDataType.Int64,
HistorianValueKind.Boolean => DriverDataType.Boolean,
HistorianValueKind.String => DriverDataType.String,
HistorianValueKind.DateTime => DriverDataType.DateTime,
_ => DefaultForEmpty,
});
}
/// <summary>
/// <see cref="IHistorianValueWriter"/> over the SQL <c>Sample</c> table. Redelivery-idempotent: rows are
/// inserted only where <c>(TagId, SourceTimeUtc)</c> is absent, so the recorder's at-least-once redelivery
@@ -36,8 +13,13 @@ internal static class SqlHistorianDataTypes
/// </summary>
/// <remarks>
/// <b>Non-finite doubles.</b> NaN / ±Infinity cannot live in a SQL <c>float</c> (the TVP throws). They are stored
/// as <see cref="HistorianValueKind.Empty"/> keeping the original status code, so the sample's presence and
/// status survive and the tag is not quarantined for one bad reading.
/// as <see cref="HistorianValueKind.Empty"/> so the sample's presence survives and the tag is not quarantined
/// for one bad reading. <b>An Empty sample is always Bad</b> (the quality contract: a stored sample is never
/// served better than "no value"): a non-finite Double whose status is not already Bad is stored with
/// <c>BadOutOfRange</c>; any other Empty with a non-Bad status is stored with <c>BadNoData</c>. An existing
/// Bad code (e.g. the comms-loss <c>BadNoCommunication</c> marker) is kept as it is.
/// <b>Tag names over <see cref="SqlServerTagCatalog.MaxNameLength"/> characters</b> are refused (the write
/// returns false, a per-tag failure for the recorder's #538 quarantine), never truncated onto another tag.
/// <b>Duplicate timestamps within a batch</b> collapse to the first (matching insert-if-absent across batches).
/// </remarks>
internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
@@ -69,6 +51,12 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
{
try
{
if (!SqlServerTagCatalog.IsValidName(tag))
{
_log.LogWarning("SQL historian live-value write refused: tag name is missing or longer than {Max} characters.",
SqlServerTagCatalog.MaxNameLength);
return false;
}
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
if (values is null || values.Count == 0) return true;
@@ -108,8 +96,9 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
}
/// <summary>
/// Normalises a batch for the TVP: null timestamp ⇒ now, non-finite doubles ⇒ Empty (status kept), and
/// duplicate timestamps collapsed to the first.
/// Normalises a batch for the TVP: null timestamp ⇒ now, non-finite doubles ⇒ Empty, every Empty forced Bad
/// (non-finite ⇒ <c>BadOutOfRange</c>, otherwise <c>BadNoData</c>, an existing Bad code kept), and duplicate
/// timestamps collapsed to the first.
/// </summary>
/// <param name="values">Input samples, in append order.</param>
/// <param name="utcNow">Clock for null timestamps.</param>
@@ -124,9 +113,17 @@ internal sealed class SqlServerHistorianValueWriter : IHistorianValueWriter
t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc);
if (!seen.Add(t.Ticks)) continue;
var value = v.Value;
var status = v.StatusCode;
if (value.Kind == HistorianValueKind.Double && !double.IsFinite(value.Number))
{
value = HistorianValue.Empty;
rows.Add(new SqlSampleRow(t, value, v.StatusCode));
if (!StatusBits.IsBad(status)) status = SqlHistorianStatusCodes.BadOutOfRange;
}
else if (value.Kind == HistorianValueKind.Empty && !StatusBits.IsBad(status))
{
status = SqlHistorianStatusCodes.BadNoData;
}
rows.Add(new SqlSampleRow(t, value, status));
}
return rows;
@@ -1,5 +1,6 @@
using System.Data;
using System.Reflection;
using System.Runtime.ExceptionServices;
using System.Text.RegularExpressions;
using Microsoft.Data.SqlClient;
using Microsoft.Extensions.Logging;
@@ -9,7 +10,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// Lazily applies the embedded, idempotent schema scripts once per instance, under a SQL application lock
/// so concurrent nodes/instances serialise. A failure is NOT latched: the next <see cref="EnsureAsync"/> retries.
/// so concurrent nodes/instances serialise. A failure is NOT latched: an <see cref="EnsureAsync"/> after
/// <see cref="FailureCacheDuration"/> retries. Within that window the last failure is re-thrown without touching
/// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one
/// semaphore and one round-trip each.
/// With <c>AutoCreateSchema=false</c> it only verifies <c>MAX(Version) &gt;= 1</c>.
/// </summary>
internal sealed partial class SqlServerSchemaBootstrapper
@@ -20,21 +24,30 @@ internal sealed partial class SqlServerSchemaBootstrapper
private const string LockResource = "OtOpcUaHistorianSchema";
private const int LockTimeoutMs = 30_000;
/// <summary>How long the last bootstrap failure is re-thrown before SQL is tried again.</summary>
public static readonly TimeSpan FailureCacheDuration = TimeSpan.FromSeconds(5);
private readonly SqlHistorianConnection _conn;
private readonly bool _autoCreate;
private readonly ILogger _log;
private readonly TimeProvider _time;
private readonly SemaphoreSlim _gate = new(1, 1);
private volatile bool _ready;
private ExceptionDispatchInfo? _lastFailure; // guarded by _gate
private DateTimeOffset _retryAfter; // guarded by _gate
/// <summary>Creates the bootstrapper.</summary>
/// <param name="conn">Connection helper.</param>
/// <param name="options">Validated options.</param>
/// <param name="log">Optional logger.</param>
public SqlServerSchemaBootstrapper(SqlHistorianConnection conn, SqlServerHistorianOptions options, ILogger? log = null)
/// <param name="timeProvider">Clock for the failure cache; system by default.</param>
public SqlServerSchemaBootstrapper(SqlHistorianConnection conn, SqlServerHistorianOptions options, ILogger? log = null,
TimeProvider? timeProvider = null)
{
_conn = conn;
_autoCreate = options.AutoCreateSchema;
_log = log ?? NullLogger.Instance;
_time = timeProvider ?? TimeProvider.System;
}
/// <summary>Ensures the schema is ready; no-op after the first success.</summary>
@@ -48,10 +61,25 @@ internal sealed partial class SqlServerSchemaBootstrapper
try
{
if (_ready) return;
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
if (_autoCreate) await ApplyAsync(conn, ct).ConfigureAwait(false);
else await VerifyAsync(conn, ct).ConfigureAwait(false);
_ready = true;
if (_lastFailure is { } cached && _time.GetUtcNow() < _retryAfter) cached.Throw();
try
{
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
if (_autoCreate) await ApplyAsync(conn, ct).ConfigureAwait(false);
else await VerifyAsync(conn, ct).ConfigureAwait(false);
_ready = true;
_lastFailure = null;
}
catch (Exception ex) when (ex is not OperationCanceledException || !ct.IsCancellationRequested)
{
// A caller's own cancellation is not a backend failure and is not cached.
_lastFailure = ExceptionDispatchInfo.Capture(ex);
_retryAfter = _time.GetUtcNow() + FailureCacheDuration;
_log.LogWarning("Historian schema bootstrap for [{Schema}] failed ({Exception}: {Message}); " +
"historian calls fail fast for {Seconds} s before retrying.",
_conn.Schema, ex.GetType().Name, ex.Message, FailureCacheDuration.TotalSeconds);
throw;
}
}
finally
{
@@ -127,8 +155,13 @@ internal sealed partial class SqlServerSchemaBootstrapper
}
catch (Exception ex)
{
// Closing the session releases a Session-owned lock anyway.
_log.LogDebug(ex, "sp_releaseapplock failed; the lock dies with the connection");
// A pooled connection keeps its server session (sp_reset_connection does not release a Session-owned
// applock), so disposing this connection would return it to the pool STILL HOLDING the lock and block
// every other node's bootstrap until that pooled session dies. Clearing the pool marks it to be
// physically closed on dispose instead, which ends the session and releases the lock.
_log.LogWarning("sp_releaseapplock failed ({Exception}); discarding the connection pool so the " +
"session holding the historian schema applock is closed rather than pooled.", ex.GetType().Name);
SqlConnection.ClearPool(conn);
}
}
@@ -6,9 +6,16 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// Tag name → <c>TagId</c> resolution shared by writes, reads, provisioning and updates. Caches positive
/// results only. Both methods run the schema bootstrap first, so callers need not.
/// <para><b>Names longer than <see cref="MaxNameLength"/></b> (the <c>Tag.Name</c> column width) are never
/// truncated: a sized <c>NVarChar(400)</c> parameter would silently cut them and resolve them onto whatever tag
/// owns the first 400 characters. <see cref="TryGetTagIdAsync"/> reports such a name as not found and
/// <see cref="GetOrCreateTagIdAsync"/> throws.</para>
/// </summary>
internal sealed class SqlServerTagCatalog
{
/// <summary>Width of the <c>Tag.Name</c> column; longer names are refused.</summary>
public const int MaxNameLength = 400;
private readonly SqlHistorianConnection _conn;
private readonly SqlServerSchemaBootstrapper _bootstrap;
private readonly ConcurrentDictionary<string, int> _cache = new(StringComparer.Ordinal);
@@ -22,12 +29,18 @@ internal sealed class SqlServerTagCatalog
_bootstrap = bootstrap;
}
/// <summary>Read-only lookup; never creates a row.</summary>
/// <summary>True when <paramref name="name"/> fits the <c>Tag.Name</c> column without truncation.</summary>
/// <param name="name">Historian tag name.</param>
/// <returns>Whether the name is non-null and at most <see cref="MaxNameLength"/> characters.</returns>
public static bool IsValidName(string? name) => name is not null && name.Length <= MaxNameLength;
/// <summary>Read-only lookup; never creates a row. A name over <see cref="MaxNameLength"/> is not found.</summary>
/// <param name="name">Historian tag name.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The TagId, or null when the tag does not exist.</returns>
public async Task<int?> TryGetTagIdAsync(string name, CancellationToken ct)
{
if (!IsValidName(name)) return null;
if (_cache.TryGetValue(name, out var cached)) return cached;
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
@@ -41,8 +54,11 @@ internal sealed class SqlServerTagCatalog
/// <param name="dataType">Initial <c>Tag.DataType</c> (only used on create; provisioning corrects it later).</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The TagId.</returns>
/// <exception cref="ArgumentException">The name is longer than <see cref="MaxNameLength"/>.</exception>
public async Task<int> GetOrCreateTagIdAsync(string name, byte dataType, CancellationToken ct)
{
if (!IsValidName(name))
throw new ArgumentException($"Historian tag name exceeds {MaxNameLength} characters.", nameof(name));
if (_cache.TryGetValue(name, out var cached)) return cached;
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false);
@@ -52,7 +68,7 @@ internal sealed class SqlServerTagCatalog
try
{
await using var cmd = _conn.CreateCommand(conn, sql);
cmd.Parameters.Add(new SqlParameter("@name", System.Data.SqlDbType.NVarChar, 400) { Value = name });
cmd.Parameters.Add(new SqlParameter("@name", System.Data.SqlDbType.NVarChar, MaxNameLength) { Value = name });
cmd.Parameters.Add(new SqlParameter("@dt", System.Data.SqlDbType.TinyInt) { Value = dataType });
await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
}
@@ -66,14 +82,10 @@ internal sealed class SqlServerTagCatalog
return id;
}
/// <summary>Drops a cached id (e.g. after a tag row is deleted).</summary>
/// <param name="name">Historian tag name.</param>
public void Invalidate(string name) => _cache.TryRemove(name, out _);
private async Task<int?> SelectIdAsync(SqlConnection conn, string name, CancellationToken ct)
{
await using var cmd = _conn.CreateCommand(conn, $"SELECT TagId FROM {_conn.Q("Tag")} WHERE Name = @name");
cmd.Parameters.Add(new SqlParameter("@name", System.Data.SqlDbType.NVarChar, 400) { Value = name });
cmd.Parameters.Add(new SqlParameter("@name", System.Data.SqlDbType.NVarChar, MaxNameLength) { Value = name });
var o = await cmd.ExecuteScalarAsync(ct).ConfigureAwait(false);
return o is null or DBNull ? null : Convert.ToInt32(o);
}
+16 -16
View File
@@ -265,30 +265,29 @@ if (hasDriver)
// Config-gated server-side HistoryRead backend. When the ServerHistorian section is enabled this
// overrides the NullHistorianDataSource default from AddOtOpcUaRuntime (last registration wins) with
// a read-only HistorianGateway-backed data source the node manager's HistoryRead overrides
// block-bridge to. The factory lives in the Gateway driver (which owns the package-client adapter
// and the ServerHistorianOptions -> client-options mapping); Runtime owns the gating.
// the selected backend's data source (Gateway or SqlServer, built by the resolved provider) that the
// node manager's HistoryRead overrides block-bridge to. Runtime owns the gating.
builder.Services.AddServerHistorian(
builder.Configuration,
(_, sp) => historianProvider.CreateDataSource(sp));
// Config-gated historian tag provisioning. When the ServerHistorian section is enabled this registers the
// gateway-backed GatewayTagProvisioner (overriding the NullHistorianProvisioning default) so that deploying
// a historized tag auto-ensures it in the historian via the gateway's EnsureTags. The AddressSpaceApplier
// (constructed in WithOtOpcUaRuntimeActors) resolves IHistorianProvisioning and fires a non-blocking
// EnsureTagsAsync for added historized tags on every deploy; without this registration the applier falls
// back to the no-op NullHistorianProvisioning and provisioning never reaches the gateway. Gated on the SAME
// ServerHistorian:Enabled flag as the read path — both target the same single gateway.
// selected backend's provisioner (GatewayTagProvisioner -> the gateway's EnsureTags, or the SqlServer Tag-row
// MERGE), overriding the NullHistorianProvisioning default, so deploying a historized tag auto-ensures it in
// the historian. The AddressSpaceApplier (constructed in WithOtOpcUaRuntimeActors) resolves
// IHistorianProvisioning and fires a non-blocking EnsureTagsAsync for added/changed historized tags on every
// deploy; without this registration the applier falls back to the no-op NullHistorianProvisioning. Gated on
// the SAME ServerHistorian:Enabled flag as the read path — both use the one resolved backend.
builder.Services.AddHistorianProvisioning(
builder.Configuration,
(_, sp) => historianProvider.CreateProvisioning(sp));
// Continuous historization of driver (non-Galaxy) tag values. Gated on ContinuousHistorization:Enabled
// AND the ServerHistorian gateway being configured: the recorder drains driver-tag live values to the
// SAME single gateway's WriteLiveValues SQL path, sourcing endpoint/key/TLS from the ServerHistorian
// section (this section carries only the recorder + outbox knobs). When both are on, register the durable
// crash-safe outbox + the gateway-backed live-value writer here; WithOtOpcUaRuntimeActors (below) spawns
// the recorder actor itself, gated on the same options.
// AND ServerHistorian:Enabled: the recorder drains driver-tag live values through the selected backend's
// value writer (the gateway's WriteLiveValues, or the SqlServer Sample table), which takes its connection
// from the ServerHistorian section (this section carries only the recorder + outbox knobs). When both are
// on, register the durable crash-safe outbox + the backend's live-value writer here;
// WithOtOpcUaRuntimeActors (below) spawns the recorder actor itself, gated on the same options.
var continuousHistorizationOptions = builder.Configuration
.GetSection(ContinuousHistorizationOptions.SectionName).Get<ContinuousHistorizationOptions>()
?? new ContinuousHistorizationOptions();
@@ -317,8 +316,9 @@ if (hasDriver)
return outbox;
});
// Gateway-backed live-value writer over its OWN gRPC channel to the same single gateway (a second
// channel to a co-located sidecar is cheap — the gateway pools the historian sessions server-side).
// The backend's live-value writer. On Gateway it opens its OWN gRPC channel to the same gateway (a second
// channel to a co-located sidecar is cheap — the gateway pools the historian sessions server-side); on
// SqlServer it shares the provider's connection helper, schema bootstrapper and tag catalog.
builder.Services.AddSingleton<IHistorianValueWriter>(sp => historianProvider.CreateValueWriter(sp));
}
@@ -205,14 +205,29 @@ internal static class HistoryPaging
/// remains, or <c>null</c> when the window is exhausted (terminal — no continuation point).</param>
/// <param name="nextSkip">The next page's boundary skip count: the running emitted-tie total while the
/// cluster drains, else <c>0</c> after advancing past it.</param>
/// <param name="reverse">
/// <c>true</c> for a reverse read (Part 11: <c>start &gt; end</c>, samples in descending time, window
/// <c>(end, start]</c>). A drained cluster then steps DOWN to <c>T − 1 tick</c>, which is lossless for the
/// same reason <c>T + 1 tick</c> is going forward, and the read continues only while that is still strictly
/// after <paramref name="endUtc"/> (the reverse window excludes <c>end</c>; a resume at exactly <c>end</c>
/// would turn into a <c>start == end</c> point read and return the excluded sample). Forward is unchanged.
/// </param>
public static void SliceTieCluster(
int clusterCount, int skip, uint cap, DateTime boundaryT, DateTime endUtc,
out int sliceStart, out int sliceCount, out DateTime? nextStartUtc, out int nextSkip)
out int sliceStart, out int sliceCount, out DateTime? nextStartUtc, out int nextSkip,
bool reverse = false)
{
sliceStart = Math.Min(skip, clusterCount);
sliceCount = Math.Min((int)cap, clusterCount - sliceStart);
var emitted = sliceStart + sliceCount;
if (emitted < clusterCount) { nextStartUtc = boundaryT; nextSkip = emitted; }
else if (reverse)
{
// DateTime.MinValue cannot step down; nothing can lie below it anyway.
var more = boundaryT > DateTime.MinValue && boundaryT.AddTicks(-1) > endUtc;
nextStartUtc = more ? boundaryT.AddTicks(-1) : null;
nextSkip = 0;
}
else
{
var next = boundaryT.AddTicks(1);
@@ -5295,6 +5295,10 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (details.IsDeleteModified) return null;
var start = details.StartTime;
var end = details.EndTime;
// Part 11 requires both bounds of a raw delete; DateTime.MinValue is the wire's "unspecified". Refuse
// it here rather than let a backend read MinValue as "from the beginning of time".
if (start == DateTime.MinValue || end == DateTime.MinValue)
return (_, _, _) => Task.FromResult(new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidArgument });
return async (updater, tagname, ct) =>
{
var code = await updater.DeleteRawAsync(tagname, start, end, ct).ConfigureAwait(false);
@@ -5898,10 +5902,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Compute in uint so +1 can never wrap: Math.Max(1,…) also guards against a zero/negative
// config value slipping through (Validate() warns but does not block startup).
var overfetchCap = (uint)Math.Max(1, MaxTieClusterOverfetch) + 1u;
var cluster = (await HistorianDataSource
IReadOnlyList<DataValueSnapshot> cluster = (await HistorianDataSource
.ReadRawAsync(tagname, startUtc, startUtc, overfetchCap, ct)
.ConfigureAwait(false)).Samples;
// A start == end point read returns the ties in the backend's FORWARD order, but a reverse read
// emitted them (and counted boundarySkip) in descending order — the mirror of forward. Slice the
// cluster in the read's own direction, or the skip would index from the wrong end and re-emit ties.
var reverse = startUtc > endUtc;
if (reverse) cluster = cluster.Reverse().ToList();
// Absurd burst: more ties than we're willing to buffer in memory. Preserve today's loud-fail
// for that node rather than over-fetch an unbounded cluster; the operator's remedy is a
// larger ServerHistorian:MaxTieClusterOverfetch (or NumValuesPerNode).
@@ -5916,9 +5926,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
return;
}
// Direction comes from the window itself (Part 11: start > end is a reverse read). A resumed reverse
// read keeps start > end, because a reverse resume point is always strictly after endUtc.
HistoryPaging.SliceTieCluster(
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip);
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
reverse: reverse);
var slice = new List<DataValueSnapshot>(sliceCount);
for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]);
@@ -196,6 +196,12 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
/// the first item that ingests cleanly — see <see cref="IngestAsync"/>.</summary>
private bool _ingestFaultLogged;
/// <summary>Whether the current run of unsupported-kind drops has already been logged, so a backend that
/// drops a whole class of value (every comms-loss marker on the Gateway backend, say) logs once per run rather
/// than once per value. The <see cref="_droppedUnsupportedKind"/> counter still counts every value. Cleared by
/// the next value that is captured.</summary>
private bool _droppedKindLogged;
/// <summary>Last observed state of <see cref="_captureGate"/>, so a change is logged once rather
/// than on every value. Seeded open: a gate that starts closed logs its first suppression.</summary>
private bool _captureGateOpen = true;
@@ -553,10 +559,18 @@ public sealed class ContinuousHistorizationRecorder : ReceiveActor, IWithTimers
// (e.g. String/Empty on the gateway's numeric-only SQL analog path) — drop and meter. No value
// content is logged.
_droppedUnsupportedKind++;
_log.Debug("ContinuousHistorization: dropped a value of a kind the historian backend does not support.");
if (!_droppedKindLogged)
{
_droppedKindLogged = true;
_log.Debug("ContinuousHistorization: dropped a value of a kind the historian backend does not support. "
+ "Further drops are only counted (DroppedUnsupportedKind) until a value is captured.");
}
return;
}
_droppedKindLogged = false;
var timestampUtc = DateTime.SpecifyKind(msg.TimestampUtc, DateTimeKind.Utc);
entries ??= new List<HistorizationOutboxEntry>();
@@ -4,12 +4,13 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Historian;
/// <summary>
/// Binds the <c>ServerHistorian</c> configuration section that gates the server-side
/// HistoryRead backend. When <see cref="Enabled"/> is <c>true</c>, <c>AddServerHistorian</c>
/// registers a read-only <c>HistorianGateway</c>-backed <c>IHistorianDataSource</c> (supplied by
/// historian. <see cref="Backend"/> selects the provider (<c>Gateway</c> by default, or <c>SqlServer</c>,
/// whose own keys live in the <c>ServerHistorian:SqlServer</c> sub-section). When <see cref="Enabled"/> is
/// <c>true</c>, <c>AddServerHistorian</c> registers that backend's <c>IHistorianDataSource</c> (supplied by
/// the Host) in place of the <c>NullHistorianDataSource</c> default; otherwise the Null default
/// survives and HistoryRead returns <c>GoodNoData</c>-empty.
/// <para>
/// The read client talks gRPC to the <c>ZB.MOM.WW.HistorianGateway</c> sidecar
/// On the Gateway backend the read client talks gRPC to the <c>ZB.MOM.WW.HistorianGateway</c> sidecar
/// (<c>historian_gateway.v1</c>) over <see cref="Endpoint"/>, authenticating with the
/// peppered-HMAC <see cref="ApiKey"/> (<c>histgw_&lt;id&gt;_&lt;secret&gt;</c>) in the
/// <c>Authorization: Bearer</c> header. This is the READ path only — there are no
@@ -24,17 +25,12 @@ public sealed class ServerHistorianOptions
public const string SectionName = "ServerHistorian";
/// <summary>
/// When <c>true</c>, the HistorianGateway read client is registered as the
/// When <c>true</c>, the selected backend's data source is registered as the
/// <c>IHistorianDataSource</c>; when <c>false</c> (the default) the no-op
/// <c>NullHistorianDataSource</c> stays in place and HistoryRead returns empty.
/// </summary>
public bool Enabled { get; init; }
/// <summary>
/// Absolute gateway endpoint URI the read client dials (e.g. <c>https://host:5222</c>).
/// The scheme selects the transport: <c>https://</c> = TLS, <c>http://</c> = h2c plaintext.
/// Required when <see cref="Enabled"/> is <c>true</c>.
/// </summary>
/// <summary>
/// Selects the historian backend provider (case-insensitive). Default <c>"Gateway"</c>; the Host's
/// <c>HistorianBackendRegistry</c> owns the set of valid names. Gateway-specific keys
@@ -49,6 +45,12 @@ public sealed class ServerHistorianOptions
public bool IsGatewayBackend =>
string.Equals(Backend?.Trim(), GatewayBackendName, StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Gateway backend only: absolute gateway endpoint URI the read client dials (e.g.
/// <c>https://host:5222</c>). The scheme selects the transport: <c>https://</c> = TLS, <c>http://</c> = h2c
/// plaintext. Required when the Gateway backend is consumed (<see cref="Enabled"/>, or an enabled
/// <c>AlarmHistorian</c>).
/// </summary>
public string Endpoint { get; init; } = "";
/// <summary>
@@ -129,17 +131,17 @@ public sealed class ServerHistorianOptions
// Gateway-specific checks apply only to the Gateway backend (other backends carry their own options).
if (IsGatewayBackend)
{
if (string.IsNullOrWhiteSpace(Endpoint))
warnings.Add("ServerHistorian:Endpoint is empty while the historian is enabled — the read client has no gateway address to dial.");
// archreview 06/S-11 warning parity: a non-empty but unparseable / non-absolute-http(s) Endpoint
// is just as fatal (it throws in the gateway client factory). Warn on it here so the Runtime-side
// registration logs describe malformed endpoints too, consistent with the Host's fail-fast
// ServerHistorianOptionsValidator (which turns this same condition into a startup failure).
else if (!Uri.TryCreate(Endpoint, UriKind.Absolute, out var uri)
|| (uri.Scheme != Uri.UriSchemeHttp && uri.Scheme != Uri.UriSchemeHttps))
warnings.Add($"ServerHistorian:Endpoint ('{Endpoint}') is not an absolute http(s) URI while the historian is enabled — the read client cannot dial it (e.g. https://host:5222).");
if (string.IsNullOrWhiteSpace(ApiKey))
warnings.Add("ServerHistorian:ApiKey is empty while the historian is enabled — the gateway gRPC surface will reject unauthenticated calls.");
if (string.IsNullOrWhiteSpace(Endpoint))
warnings.Add("ServerHistorian:Endpoint is empty while the historian is enabled — the read client has no gateway address to dial.");
// archreview 06/S-11 warning parity: a non-empty but unparseable / non-absolute-http(s) Endpoint
// is just as fatal (it throws in the gateway client factory). Warn on it here so the Runtime-side
// registration logs describe malformed endpoints too, consistent with the Host's fail-fast
// ServerHistorianOptionsValidator (which turns this same condition into a startup failure).
else if (!Uri.TryCreate(Endpoint, UriKind.Absolute, out var uri)
|| (uri.Scheme != Uri.UriSchemeHttp && uri.Scheme != Uri.UriSchemeHttps))
warnings.Add($"ServerHistorian:Endpoint ('{Endpoint}') is not an absolute http(s) URI while the historian is enabled — the read client cannot dial it (e.g. https://host:5222).");
if (string.IsNullOrWhiteSpace(ApiKey))
warnings.Add("ServerHistorian:ApiKey is empty while the historian is enabled — the gateway gRPC surface will reject unauthenticated calls.");
}
// MaxTieClusterOverfetch is intentionally checked AFTER the Enabled early-return above:
// the over-fetch code path only runs when a real IHistorianDataSource is wired in,
@@ -130,4 +130,79 @@ public class HistoryUpdateCommandTests
await Should.ThrowAsync<CommandException>(async () => await cmd.ExecuteAsync(console));
TestConsoleHelper.GetOutput(console).ShouldContain("BadEntryExists");
}
private static readonly string[] TwoTimes = ["2026-01-15T08:00:00Z", "2026-01-15T09:00:00Z"];
/// <summary>A Good service result that carries NO per-item results must not exit 0.</summary>
[Fact]
public async Task Execute_GoodServiceResult_WithMissingItemResults_Throws()
{
var svc = new FakeOpcUaClientService
{
HistoryUpdateResult = new HistoryUpdateOutcome(StatusCodes.Good, [StatusCodes.Good], [new List<StatusCode>()])
};
using var console = TestConsoleHelper.CreateConsole();
await Should.ThrowAsync<CommandException>(async () =>
await Make(svc, values: ["1", "2"], times: TwoTimes).ExecuteAsync(console));
TestConsoleHelper.GetOutput(console).ShouldContain("(no result returned)");
}
/// <summary>Fewer per-item results than items sent must not exit 0, even when every returned one is Good.</summary>
[Fact]
public async Task Execute_TooFewItemResults_Throws()
{
var svc = new FakeOpcUaClientService
{
HistoryUpdateResult = new HistoryUpdateOutcome(StatusCodes.Good, [StatusCodes.Good],
[new List<StatusCode> { StatusCodes.GoodEntryInserted }])
};
using var console = TestConsoleHelper.CreateConsole();
await Should.ThrowAsync<CommandException>(async () =>
await Make(svc, values: ["1", "2"], times: TwoTimes).ExecuteAsync(console));
}
/// <summary>A Good service result with no history-update result at all must not exit 0.</summary>
[Fact]
public async Task Execute_NoResultEntry_Throws()
{
var svc = new FakeOpcUaClientService
{
HistoryUpdateResult = new HistoryUpdateOutcome(StatusCodes.Good, [], [])
};
using var console = TestConsoleHelper.CreateConsole();
await Should.ThrowAsync<CommandException>(async () =>
await Make(svc, mode: "delete-raw", start: TwoTimes[0], end: TwoTimes[1]).ExecuteAsync(console));
}
/// <summary>An Uncertain item result is not Good and fails the command.</summary>
[Fact]
public async Task Execute_UncertainItem_Throws()
{
var svc = new FakeOpcUaClientService
{
HistoryUpdateResult = new HistoryUpdateOutcome(StatusCodes.Good, [StatusCodes.Good],
[new List<StatusCode> { StatusCodes.UncertainDataSubNormal }])
};
using var console = TestConsoleHelper.CreateConsole();
await Should.ThrowAsync<CommandException>(async () =>
await Make(svc, values: ["1"], times: ["2026-01-15T08:00:00Z"]).ExecuteAsync(console));
TestConsoleHelper.GetOutput(console).ShouldContain("UncertainDataSubNormal");
}
/// <summary>A status the SDK has no name for prints as hex, never blank.</summary>
[Fact]
public async Task Execute_UnnamedStatus_PrintsHex()
{
const uint unnamed = 0x8FFF0000u;
StatusCodes.GetBrowseName(unnamed).ShouldBeNullOrEmpty(); // precondition: genuinely unnamed
var svc = new FakeOpcUaClientService
{
HistoryUpdateResult = new HistoryUpdateOutcome(StatusCodes.Good, [StatusCodes.Good],
[new List<StatusCode> { new(unnamed) }])
};
using var console = TestConsoleHelper.CreateConsole();
await Should.ThrowAsync<CommandException>(async () =>
await Make(svc, values: ["1"], times: ["2026-01-15T08:00:00Z"]).ExecuteAsync(console));
TestConsoleHelper.GetOutput(console).ShouldContain("0x8FFF0000");
}
}
@@ -73,6 +73,24 @@ public sealed class GatewayHistorianValueWriterTests
Assert.False(ok);
}
[Fact]
public async Task All_non_numeric_batch_is_acked_without_calling_the_gateway()
{
// A soft-failure ack: had the writer called the gateway, the result would be false.
var fake = new FakeHistorianGatewayClient { WriteLiveValuesResult = new WriteAck { Success = false, Queued = false } };
var values = new[]
{
new HistorizationValue(DateTime.UtcNow, HistorianValue.FromString("text"), 0u),
new HistorizationValue(DateTime.UtcNow, HistorianValue.FromDateTime(DateTime.UtcNow), 0u),
new HistorizationValue(DateTime.UtcNow, HistorianValue.Empty, 0x80310000u),
};
var ok = await Writer(fake).WriteLiveValuesAsync("T", values, TestContext.Current.CancellationToken);
Assert.True(ok);
Assert.Equal(0, fake.WriteLiveValuesCallCount);
}
[Fact]
public async Task Empty_batch_is_trivially_acked_without_calling_the_gateway()
{
@@ -116,6 +116,19 @@ public sealed class AtTimeReadTests(SqlHistorianFixture fx)
s.StatusCode.ShouldBe(GoodInterpolated);
}
[Fact]
public async Task A_time_after_now_stepped_from_the_last_sample_is_uncertain()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync(); // last sample +40 = 50, Good
var src = db.Source(new FixedClock(new DateTimeOffset(At(42))));
var s = (await src.ReadAtTimeAsync("A", [At(41), At(45)], Ct)).Samples;
s[0].Value.ShouldBe(50d);
s[0].StatusCode.ShouldBe(GoodInterpolated); // after the last sample but not after now: held Good
s[1].Value.ShouldBe(50d);
s[1].StatusCode.ShouldBe(UncertainInterpolated); // after now: the value is not known to still hold
}
[Fact]
public async Task An_uncertain_neighbour_makes_the_interpolation_uncertain()
{
@@ -0,0 +1,9 @@
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
/// <summary>A settable clock for the data source's "now" and the bootstrapper's failure cache.</summary>
internal sealed class FixedClock(DateTimeOffset now) : TimeProvider
{
public DateTimeOffset Now { get; set; } = now;
public override DateTimeOffset GetUtcNow() => Now;
}
@@ -0,0 +1,87 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
/// <summary>
/// Names longer than the 400-character <c>Tag.Name</c> / <c>Event.SourceName</c> columns are refused, never
/// truncated. Each test seeds a series under the 400-character PREFIX of the long name: a sized
/// <c>NVarChar(400)</c> parameter would silently cut the long name to that prefix and hit the wrong series.
/// </summary>
[Collection(SqlHistorianCollection.Name)]
public sealed class LongNameTests(SqlHistorianFixture fx)
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static DateTime At(double s) => ReadTestDb.At(s);
private const uint BadInvalidArgument = 0x80AB0000u;
private const uint BadNoData = 0x809B0000u;
private static readonly string Prefix = new('p', 400);
private static readonly string Long = Prefix + "-x";
[Fact]
public async Task The_writer_refuses_a_long_name_as_a_per_tag_failure_and_writes_nothing()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good));
var w = new SqlServerHistorianValueWriter(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot));
(await w.WriteLiveValuesAsync(Long, [new(At(5), HistorianValue.FromDouble(2), 0u)], Ct)).ShouldBeFalse();
(await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Sample")}")).ShouldBe(1);
(await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Tag")}")).ShouldBe(1);
}
[Fact]
public async Task The_updater_answers_BadInvalidArgument_for_every_item_of_a_long_name()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good));
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot));
(await u.UpdateDataAsync(Long, HistoryUpdateMode.Update,
[new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], Ct))
.ShouldBe([BadInvalidArgument, BadInvalidArgument]);
(await u.DeleteRawAsync(Long, At(-1), At(10), Ct)).ShouldBe(BadInvalidArgument);
(await u.DeleteAtTimeAsync(Long, [At(0)], Ct)).ShouldBe([BadInvalidArgument]);
// The prefix series is untouched: not replaced, not deleted.
var raw = (await db.Source().ReadRawAsync(Prefix, At(-1), At(10), 0, Ct)).Samples;
raw.Select(s => s.Value).ShouldBe([(object)1d]);
}
[Fact]
public async Task Reads_of_a_long_name_are_not_found_rather_than_the_prefix_series()
{
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAsync(Prefix, (At(0), HistorianValue.FromDouble(1), ReadTestDb.Good));
var src = db.Source();
(await src.ReadRawAsync(Long, At(-1), At(10), 0, Ct)).Samples.ShouldBeEmpty();
(await src.ReadAtTimeAsync(Long, [At(0)], Ct)).Samples.Single().StatusCode.ShouldBe(BadNoData);
(await src.ReadProcessedAsync(Long, At(-1), At(10), TimeSpan.Zero, HistoryAggregateType.Count, Ct))
.Samples.Single().StatusCode.ShouldBe(HistorianInfoBits.WithCalculated(BadNoData));
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task An_event_read_filtered_by_a_long_source_matches_nothing()
{
var db = await ReadTestDb.CreateAsync(fx);
await using (var c = await db.Conn.OpenAsync(Ct))
await using (var cmd = db.Conn.CreateCommand(c,
$"INSERT {db.Conn.Q("Event")} (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, " +
"UserName, Comment, EventTimeUtc, ReceivedTimeUtc) VALUES ('e1', @src, 'a', 'n', 'T', 500, 'Activated', NULL, NULL, NULL, @t, @t)"))
{
cmd.Parameters.AddWithValue("@src", Prefix);
SqlHistorianConnection.AddDateTime2(cmd, "@t", At(0));
await cmd.ExecuteNonQueryAsync(Ct);
}
var src = db.Source();
(await src.ReadEventsAsync(Prefix, At(-1), At(10), 0, Ct)).Events.Count.ShouldBe(1); // control
(await src.ReadEventsAsync(Long, At(-1), At(10), 0, Ct)).Events.ShouldBeEmpty();
}
}
@@ -187,7 +187,7 @@ public sealed class ProcessedReadTests(SqlHistorianFixture fx)
public async Task A_weekly_interval_over_four_centuries_lands_samples_in_the_right_week()
{
// Hand-computed (calendar arithmetic, independent of SQL): 1601-01-01 → 2027-01-01 is 155 593 days =
// 22 227.57 weeks ⇒ 22 228 intervals (under the 100k cap). 2026-10-09 08:00 is 155 498 days in ⇒ week
// 22 227.57 weeks ⇒ 22 228 intervals (under the 100k cap). 2026-10-09 08:00 is 155 509 days in ⇒ week
// index 22 215, which starts Monday 2026-10-05 00:00; every fixture sample (t0 … t0+40 s) is in it.
var db = await ReadTestDb.CreateAsync(fx);
await db.SeedAAsync();
@@ -0,0 +1,104 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
/// <summary>
/// Final-review I-1: a stored sample is never served with a better quality than "no value". An Empty (null-value)
/// sample is always Bad — forced Bad by the value writer, refused non-Bad by the updater, and clamped to
/// <c>BadNoData</c> on read for rows written before either rule existed.
/// </summary>
[Collection(SqlHistorianCollection.Name)]
public sealed class QualityContractTests(SqlHistorianFixture fx)
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static DateTime At(double s) => ReadTestDb.At(s);
private const uint BadOutOfRange = 0x803C0000u;
private const uint BadNoData = 0x809B0000u;
private const uint BadInvalidArgument = 0x80AB0000u;
private const uint BadNoCommunication = 0x80310000u;
private static bool IsBad(uint status) => (status & 0x80000000u) != 0;
private static SqlServerHistorianValueWriter Writer(ReadTestDb db) =>
new(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot));
private static SqlServerHistorianDataUpdater Updater(ReadTestDb db) =>
new(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot));
[Fact]
public async Task A_good_NaN_written_through_the_real_writer_reads_back_Bad_on_Raw_and_AtTime()
{
var db = await ReadTestDb.CreateAsync(fx);
(await Writer(db).WriteLiveValuesAsync("nan", [
new(At(0), HistorianValue.FromDouble(10), 0u),
new(At(10), HistorianValue.FromDouble(double.NaN), 0u),
new(At(20), HistorianValue.FromDouble(double.PositiveInfinity), 0x40000000u),
new(At(30), HistorianValue.FromDouble(double.NegativeInfinity), BadNoCommunication),
], Ct)).ShouldBeTrue();
var raw = (await db.Source().ReadRawAsync("nan", At(0), At(100), 0, Ct)).Samples;
raw.Count.ShouldBe(4);
raw[1].Value.ShouldBeNull();
raw[1].StatusCode.ShouldBe(BadOutOfRange); // Good NaN ⇒ stored Bad
raw[2].StatusCode.ShouldBe(BadOutOfRange); // Uncertain +Inf ⇒ stored Bad
raw[3].StatusCode.ShouldBe(BadNoCommunication); // an existing Bad code is kept
var at = (await db.Source().ReadAtTimeAsync("nan", [At(10), At(15)], Ct)).Samples;
at.ShouldAllBe(s => s.Value == null);
at.ShouldAllBe(s => IsBad(s.StatusCode));
at[1].StatusCode.ShouldBe(BadOutOfRange); // inside the "outage" the NaN opened
}
[Fact]
public async Task A_non_bad_Empty_written_through_the_real_writer_is_stored_BadNoData()
{
var db = await ReadTestDb.CreateAsync(fx);
(await Writer(db).WriteLiveValuesAsync("empty", [
new(At(0), HistorianValue.Empty, 0u),
new(At(1), HistorianValue.Empty, 0x40000000u),
new(At(2), HistorianValue.Empty, BadNoCommunication),
], Ct)).ShouldBeTrue();
// Read the stored column directly so the read-side clamp cannot mask a writer that forgot to force Bad.
(await db.ScalarAsync($"SELECT COUNT(*) FROM {db.Conn.Q("Sample")} WHERE StatusCode >= 0")).ShouldBe(0);
var raw = (await db.Source().ReadRawAsync("empty", At(0), At(10), 0, Ct)).Samples;
raw.Select(s => s.StatusCode).ShouldBe([BadNoData, BadNoData, BadNoCommunication]);
}
[Fact]
public async Task A_legacy_good_Empty_row_is_clamped_to_BadNoData_on_Raw_and_AtTime()
{
var db = await ReadTestDb.CreateAsync(fx);
// Seeded with plain INSERTs, bypassing the writer: what a pre-fix build could have stored.
await db.SeedAsync("legacy",
(At(0), HistorianValue.FromDouble(10), ReadTestDb.Good),
(At(10), HistorianValue.Empty, ReadTestDb.Good),
(At(20), HistorianValue.Empty, ReadTestDb.Uncertain),
(At(30), HistorianValue.FromDouble(40), ReadTestDb.Good));
var raw = (await db.Source().ReadRawAsync("legacy", At(0), At(100), 0, Ct)).Samples;
raw.Select(s => s.StatusCode).ShouldBe([ReadTestDb.Good, BadNoData, BadNoData, ReadTestDb.Good]);
var at = (await db.Source().ReadAtTimeAsync("legacy", [At(10), At(15), At(25)], Ct)).Samples;
at.ShouldAllBe(s => s.Value == null && s.StatusCode == BadNoData);
}
[Fact]
public async Task The_updater_refuses_a_non_bad_Empty_and_accepts_a_bad_one()
{
var db = await ReadTestDb.CreateAsync(fx);
var r = await Updater(db).UpdateDataAsync("u", HistoryUpdateMode.Insert, [
new(At(0), HistorianValue.Empty, 0u),
new(At(1), HistorianValue.Empty, 0x40000000u),
new(At(2), HistorianValue.Empty, BadNoCommunication),
new(At(3), HistorianValue.FromDouble(1), 0u),
], Ct);
r.ShouldBe([BadInvalidArgument, BadInvalidArgument, SqlHistorianStatusCodes.GoodEntryInserted,
SqlHistorianStatusCodes.GoodEntryInserted]);
(await db.Source().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples
.Select(s => s.StatusCode).ShouldBe([BadNoCommunication, 0u]);
}
}
@@ -42,8 +42,9 @@ internal sealed class ReadTestDb
return new ReadTestDb(o, conn, boot);
}
/// <summary>A fresh data source (fresh catalog cache) over this schema.</summary>
public SqlServerHistorianDataSource Source() => new(Conn, Boot, new SqlServerTagCatalog(Conn, Boot));
/// <summary>A fresh data source (fresh catalog cache) over this schema; <paramref name="clock"/> is its "now".</summary>
public SqlServerHistorianDataSource Source(TimeProvider? clock = null) =>
new(Conn, Boot, new SqlServerTagCatalog(Conn, Boot), timeProvider: clock);
/// <summary>Inserts a tag row and the given samples. ServerTimeUtc = source time + 1 s (so tests can tell them apart).</summary>
public async Task<int> SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples)
@@ -237,7 +238,8 @@ public sealed class RawReadTests(SqlHistorianFixture fx)
ConnectionString = fx.ConnectionString, Schema = SqlHistorianFixture.UniqueSchema(), AutoCreateSchema = false,
};
var conn = new SqlHistorianConnection(o);
var boot = new SqlServerSchemaBootstrapper(conn, o);
var clock = new FixedClock(DateTimeOffset.UtcNow);
var boot = new SqlServerSchemaBootstrapper(conn, o, timeProvider: clock);
var src = new SqlServerHistorianDataSource(conn, boot, new SqlServerTagCatalog(conn, boot));
await Should.ThrowAsync<HistorianSchemaMissingException>(() => src.ReadRawAsync("A", At(0), At(1), 0, Ct));
@@ -251,6 +253,7 @@ public sealed class RawReadTests(SqlHistorianFixture fx)
// Another node creates the schema; the same data source then succeeds (bootstrap failure is not latched).
var creator = new SqlServerHistorianOptions { ConnectionString = o.ConnectionString, Schema = o.Schema };
await new SqlServerSchemaBootstrapper(new SqlHistorianConnection(creator), creator).EnsureAsync(Ct);
clock.Now += SqlServerSchemaBootstrapper.FailureCacheDuration; // past the 5 s failure cache (M-2)
(await src.ReadRawAsync("A", At(0), At(1), 0, Ct)).Samples.ShouldBeEmpty();
src.GetHealthSnapshot().ConsecutiveFailures.ShouldBe(0);
}
@@ -77,10 +77,37 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
{
fx.SkipIfUnavailable();
var s = SqlHistorianFixture.UniqueSchema();
var checker = Boot(Opts(s, auto: false));
var clock = new FixedClock(DateTimeOffset.UtcNow);
var o = Opts(s, auto: false);
var checker = new SqlServerSchemaBootstrapper(new SqlHistorianConnection(o), o, timeProvider: clock);
await Should.ThrowAsync<HistorianSchemaMissingException>(() => checker.EnsureAsync(CancellationToken.None));
await Boot(Opts(s)).EnsureAsync(CancellationToken.None);
await checker.EnsureAsync(CancellationToken.None); // same instance retries
clock.Now += SqlServerSchemaBootstrapper.FailureCacheDuration;
await checker.EnsureAsync(CancellationToken.None); // same instance retries once the failure cache expires
}
/// <summary>
/// M-2: within <see cref="SqlServerSchemaBootstrapper.FailureCacheDuration"/> of a failure the bootstrapper
/// re-throws it without touching SQL — proven by creating the schema in between: a call that went to SQL
/// would now pass.
/// </summary>
[Fact]
public async Task A_bootstrap_failure_is_cached_for_five_seconds_then_retried()
{
fx.SkipIfUnavailable();
var s = SqlHistorianFixture.UniqueSchema();
var clock = new FixedClock(DateTimeOffset.UtcNow);
var o = Opts(s, auto: false);
var checker = new SqlServerSchemaBootstrapper(new SqlHistorianConnection(o), o, timeProvider: clock);
await Should.ThrowAsync<HistorianSchemaMissingException>(() => checker.EnsureAsync(CancellationToken.None));
await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // the schema now exists
clock.Now += SqlServerSchemaBootstrapper.FailureCacheDuration - TimeSpan.FromTicks(1);
await Should.ThrowAsync<HistorianSchemaMissingException>(() => checker.EnsureAsync(CancellationToken.None));
clock.Now += TimeSpan.FromTicks(1);
await checker.EnsureAsync(CancellationToken.None);
}
[Fact]
@@ -89,7 +89,7 @@ public sealed class WriteTests(SqlHistorianFixture fx)
}
[Fact]
public async Task Non_finite_doubles_store_as_Empty_with_their_status_and_do_not_fail_the_write()
public async Task Non_finite_doubles_store_as_Empty_forced_Bad_and_do_not_fail_the_write()
{
fx.SkipIfUnavailable();
var (w, s) = Make();
@@ -101,7 +101,8 @@ public sealed class WriteTests(SqlHistorianFixture fx)
rows.Count.ShouldBe(3);
rows[0].Kind.ShouldBe((byte)HistorianValueKind.Empty);
rows[0].N.ShouldBeNull();
rows[0].Status.ShouldBe(unchecked((int)0x40000000u));
rows[0].Status.ShouldBe(unchecked((int)SqlHistorianStatusCodes.BadOutOfRange)); // Uncertain NaN ⇒ Bad (I-1)
rows[1].Status.ShouldBe(unchecked((int)SqlHistorianStatusCodes.BadOutOfRange)); // Good +Inf ⇒ Bad (I-1)
rows[2].N.ShouldBe(7);
}
@@ -32,6 +32,36 @@ public sealed class SqlServerHistorianProviderTests
p.CreateUpdater(sp).ShouldBeOfType<SqlServerHistorianDataUpdater>();
}
[Fact]
public void Supported_kinds_are_a_frozen_set() =>
Make().SupportedKinds.ShouldBeAssignableTo<System.Collections.Frozen.FrozenSet<HistorianValueKind>>();
/// <summary>
/// An invalid schema must not fail provider construction (the host resolves the provider even when no
/// consumer is enabled); it surfaces only when a seam is actually built.
/// </summary>
[Fact]
public void An_invalid_schema_fails_only_when_a_seam_is_built()
{
var p = new SqlServerHistorianProvider(new SqlServerHistorianOptions { ConnectionString = "x", Schema = "bad;schema" });
p.Name.ShouldBe("SqlServer");
Should.Throw<ArgumentException>(() => p.CreateDataSource(new NullSp()));
}
[Fact]
public void Every_seam_shares_one_connection_bootstrapper_and_catalog()
{
var sp = new NullSp();
var p = Make();
var a = (SqlServerHistorianDataSource)p.CreateDataSource(sp);
var b = (SqlServerHistorianDataSource)p.CreateDataSource(sp);
Field(a, "_bootstrap").ShouldBeSameAs(Field(b, "_bootstrap"));
Field(a, "_catalog").ShouldBeSameAs(Field(b, "_catalog"));
}
private static object? Field(object o, string name) =>
o.GetType().GetField(name, System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic)!.GetValue(o);
private sealed class NullSp : IServiceProvider
{
public object? GetService(Type serviceType) => null;
@@ -22,6 +22,23 @@ public sealed class StatusCodeOracleTests
((uint)Opc.Ua.AggregateBits.Interpolated).ShouldBe(HistorianInfoBits.HistorianInterpolated);
}
/// <summary>
/// A stored status that already carries a historian bit (a sample written via HistoryUpdate with the Calculated
/// bit, say) must come back with ONE historian bit, never both: the existing pair is masked off first. The SDK
/// agrees — <c>SetAggregateBits</c> replaces, it does not OR.
/// </summary>
[Fact]
public void Setting_one_historian_bit_clears_the_other()
{
var calculatedUncertain = SqlHistorianStatusCodes.UncertainDataSubNormal | 0x400u | HistorianInfoBits.HistorianCalculated;
var interpolated = HistorianInfoBits.WithInterpolated(calculatedUncertain);
interpolated.ShouldBe(SqlHistorianStatusCodes.UncertainDataSubNormal | 0x400u | HistorianInfoBits.HistorianInterpolated);
new Opc.Ua.StatusCode(interpolated).AggregateBits.ShouldBe(Opc.Ua.AggregateBits.Interpolated);
var interpolatedGood = 0x400u | HistorianInfoBits.HistorianInterpolated;
HistorianInfoBits.WithCalculated(interpolatedGood).ShouldBe(0x400u | HistorianInfoBits.HistorianCalculated);
}
[Fact]
public void Info_type_is_data_value() =>
HistorianInfoBits.InfoTypeDataValue.ShouldBe(0x400u);
@@ -9,18 +9,32 @@ public sealed class ValueWriterPrepareTests
private static readonly DateTime T = new(2026, 10, 9, 1, 2, 3, DateTimeKind.Utc);
[Fact]
public void Non_finite_doubles_become_Empty_keeping_status()
public void Non_finite_doubles_become_Empty_and_are_forced_Bad()
{
const uint status = 0x40000000u;
const uint uncertain = 0x40000000u;
const uint badNoCommunication = 0x80310000u;
var rows = SqlServerHistorianValueWriter.Prepare(
[
new(T, HistorianValue.FromDouble(double.NaN), status),
new(T, HistorianValue.FromDouble(double.NaN), uncertain),
new(T.AddTicks(1), HistorianValue.FromDouble(double.PositiveInfinity), 0),
new(T.AddTicks(2), HistorianValue.FromDouble(double.NegativeInfinity), 0),
new(T.AddTicks(2), HistorianValue.FromDouble(double.NegativeInfinity), badNoCommunication),
new(T.AddTicks(3), HistorianValue.FromDouble(1.5), 0),
], () => T);
rows.Select(r => r.Value.Kind).ShouldBe([HistorianValueKind.Empty, HistorianValueKind.Empty, HistorianValueKind.Empty, HistorianValueKind.Double]);
rows[0].StatusCode.ShouldBe(status);
rows.Select(r => r.StatusCode).ShouldBe([SqlHistorianStatusCodes.BadOutOfRange, SqlHistorianStatusCodes.BadOutOfRange,
badNoCommunication, 0u]);
}
[Fact]
public void A_non_bad_Empty_is_stored_BadNoData_and_a_bad_Empty_keeps_its_code()
{
var rows = SqlServerHistorianValueWriter.Prepare(
[
new(T, HistorianValue.Empty, 0),
new(T.AddTicks(1), HistorianValue.Empty, 0x40A40000u),
new(T.AddTicks(2), HistorianValue.Empty, 0x80310000u),
], () => T);
rows.Select(r => r.StatusCode).ShouldBe([SqlHistorianStatusCodes.BadNoData, SqlHistorianStatusCodes.BadNoData, 0x80310000u]);
}
[Fact]
@@ -339,4 +339,40 @@ public sealed class ServerHistorianOptionsValidatorTests
[Fact]
public void SqlServer_backend_not_consumed_passes_without_connection_string() =>
SqlSut(null).Validate(null, new ServerHistorianOptions { Backend = "SqlServer" }).Failed.ShouldBeFalse();
private static IConfiguration BadSchemaConfig(bool alarmHistorianEnabled) =>
new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
{
["AlarmHistorian:Enabled"] = alarmHistorianEnabled ? "true" : "false",
["ServerHistorian:SqlServer:ConnectionString"] = "Server=localhost;Database=h",
["ServerHistorian:SqlServer:Schema"] = "bad;schema",
}).Build();
/// <summary>
/// An invalid <c>ServerHistorian:SqlServer:Schema</c> must not fail the host when nothing consumes the section:
/// <see cref="HistorianBackendRegistry.Resolve"/> (which Program.cs calls unconditionally) does not throw and
/// the validator does not check it — the same consumer gate the connection string has.
/// </summary>
[Fact]
public void SqlServer_invalid_schema_not_consumed_neither_resolve_nor_validation_fails()
{
var configuration = BadSchemaConfig(alarmHistorianEnabled: false);
var options = new ServerHistorianOptions { Backend = "SqlServer" };
Should.NotThrow(() => HistorianBackendRegistry.Resolve(options, configuration));
new ServerHistorianOptionsValidator(configuration).Validate(null, options).Failed.ShouldBeFalse();
}
/// <summary>Consumed (here by AlarmHistorian alone), the same invalid schema is a named validator error.</summary>
[Fact]
public void SqlServer_invalid_schema_consumed_is_a_validator_error_naming_the_key()
{
var configuration = BadSchemaConfig(alarmHistorianEnabled: true);
var options = new ServerHistorianOptions { Backend = "SqlServer" };
Should.NotThrow(() => HistorianBackendRegistry.Resolve(options, configuration));
var result = new ServerHistorianOptionsValidator(configuration).Validate(null, options);
result.Failed.ShouldBeTrue();
result.Failures.ShouldContain(f => f.Contains("ServerHistorian:SqlServer:Schema"));
}
}
@@ -6,6 +6,8 @@ using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Authorization;
using ZB.MOM.WW.OtOpcUa.Host.OpcUa;
using ZB.MOM.WW.OtOpcUa.OpcUaServer.Security;
using CoreHistoryReadResult = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
using static ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.ServerEventRig;
@@ -104,6 +106,47 @@ public sealed class HistoryUpdateOverTheWireTests
finally { await rig.DisposeAsync(); }
}
/// <summary>
/// ACL flag OFF over the wire: a key's grants collapse into server-wide roles at activation (the production
/// <see cref="ApiKeyInterimRoleMapper"/>). A key whose principal holds only a Cluster-scoped WriteOperate grant
/// is denied — live-write authority never implies rewriting history — and the same key with a Cluster-scoped
/// HistoryUpdate grant is admitted. The allow half keeps the deny from passing against a build that refuses
/// every HistoryUpdate with the flag off.
/// </summary>
[Fact]
public async Task Acl_off_a_cluster_history_update_grant_allows_and_write_operate_alone_is_denied()
{
var ct = TestContext.Current.CancellationToken;
var store = new SharedStore();
var rig = await StartAsync("HuWireAclOff", encrypted: false, ct);
try
{
rig.Nm.HistorianDataUpdater = store;
rig.Nm.HistorianDataSource = store;
var id = rig.EnsureRawVariable("Speed", "Double", writable: false, Tagname);
var writeOnly = ClusterGrant(KeyPrincipal, "acl-writer", NodePermissions.Browse | NodePermissions.Read | NodePermissions.WriteOperate);
rig.Enforce(false, [Key()], writeOnly);
rig.KeyRoles = ApiKeyInterimRoleMapper.Map([writeOnly], KeyPrincipal);
rig.KeyRoles.ShouldNotContain(OpcUaDataPlaneRoles.HistoryUpdate);
using (var writer = await rig.ConnectAsync(anonymous: false, ct))
{
var denial = await UpdateAsync(writer, RawNode(writer, id), PerformUpdateType.Insert, ct, (T1, 1.0));
denial.StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied);
}
store.UpdateCalls.ShouldBe(0, "a denied session never reaches the updater");
var editor = ClusterGrant(KeyPrincipal, "acl-editor", NodePermissions.Browse | NodePermissions.Read | NodePermissions.HistoryUpdate);
rig.Enforce(false, [Key()], editor);
rig.KeyRoles = ApiKeyInterimRoleMapper.Map([editor], KeyPrincipal);
using var allowed = await rig.ConnectAsync(anonymous: false, ct);
var ok = await UpdateAsync(allowed, RawNode(allowed, id), PerformUpdateType.Insert, ct, (T1, 1.0));
ok.OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.GoodEntryInserted]);
store.UpdateCalls.ShouldBe(1);
}
finally { await rig.DisposeAsync(); }
}
/// <summary>The HistoryWrite bit is visible on both AccessLevel and UserAccessLevel when an updater is wired.</summary>
[Fact]
public async Task The_history_write_bit_is_visible_over_the_wire()
@@ -46,6 +46,13 @@ internal sealed class ServerEventRig : IAsyncDisposable
public OtOpcUaNodeManager Nm => _server.NodeManager!;
/// <summary>
/// The roles the <c>apikey</c> session activates with (read at each activation). Defaults to the two
/// server-wide roles; a flag-off test sets it from <c>ApiKeyInterimRoleMapper.Map</c> over its own grants, as
/// the production authenticator does.
/// </summary>
public IReadOnlyList<string> KeyRoles { get; set; } = [OpcUaDataPlaneRoles.AlarmAck, OpcUaDataPlaneRoles.WriteOperate];
public SdkAddressSpaceSink Sink { get; private set; } = null!;
public static async Task<ServerEventRig> StartAsync(string name, bool encrypted, CancellationToken ct)
@@ -83,7 +90,7 @@ internal sealed class ServerEventRig : IAsyncDisposable
}
private OpcUaApplicationHost NewHost() =>
new(_hostOptions, NullLogger<OpcUaApplicationHost>.Instance, new KeyPresentingAuthenticator());
new(_hostOptions, NullLogger<OpcUaApplicationHost>.Instance, new KeyPresentingAuthenticator(() => KeyRoles));
/// <summary>
/// Stops the server and boots a fresh one on the SAME port, application URI and PKI store (a process
@@ -300,18 +307,17 @@ internal sealed class ServerEventRig : IAsyncDisposable
}
/// <summary>
/// Admits <c>apikey</c> as <see cref="KeyPrincipal"/> carrying the two server-wide roles (used only
/// while enforcement is off); anonymous falls through to the <c>anonymous</c> principal.
/// Admits <c>apikey</c> as <see cref="KeyPrincipal"/> carrying <see cref="KeyRoles"/> (used only while
/// enforcement is off); anonymous falls through to the <c>anonymous</c> principal.
/// </summary>
private sealed class KeyPresentingAuthenticator : IOpcUaUserAuthenticator
private sealed class KeyPresentingAuthenticator(Func<IReadOnlyList<string>> roles) : IOpcUaUserAuthenticator
{
/// <inheritdoc />
public Task<OpcUaUserAuthResult> AuthenticateUserNameAsync(string username, string password, CancellationToken ct) =>
Task.FromResult(
string.Equals(username, "apikey", StringComparison.Ordinal)
&& password.StartsWith($"otua_{KeyId}_", StringComparison.Ordinal)
? OpcUaUserAuthResult.AllowKey(
KeyPrincipal, [OpcUaDataPlaneRoles.AlarmAck, OpcUaDataPlaneRoles.WriteOperate], KeyId)
? OpcUaUserAuthResult.AllowKey(KeyPrincipal, roles(), KeyId)
: OpcUaUserAuthResult.Deny("Invalid API key."));
}
}
@@ -166,6 +166,35 @@ public sealed class HistoryUpdateTests : IDisposable
finally { await host.DisposeAsync(); }
}
/// <summary>
/// ACL off: Anonymous is an ordinary principal, not a bypass — an anonymous session WITHOUT the HistoryUpdate
/// role (even one carrying WriteOperate) is denied, while one whose grants confer the role is admitted. The
/// allow half keeps the deny from passing against a build that denies anonymous unconditionally.
/// </summary>
[Fact]
public async Task Acl_off_anonymous_without_the_role_is_denied_and_with_it_is_allowed()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var denied = Insert(nm, AnonymousIdentity(OpcUaDataPlaneRoles.WriteOperate), nodeId, Value(T1, 1d));
denied.Errors[0].StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied);
denied.Results[0].StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied);
updater.UpdateCalls.ShouldBeEmpty();
var allowed = Insert(nm, AnonymousIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(T2, 2d));
allowed.Errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good);
updater.UpdateCalls.Count.ShouldBe(1);
}
finally { await host.DisposeAsync(); }
}
// ───────────────────────────── availability + AccessLevel ─────────────────────────────
/// <summary>
@@ -241,6 +270,82 @@ public sealed class HistoryUpdateTests : IDisposable
finally { await host.DisposeAsync(); }
}
/// <summary>
/// A surgical deploy that turns historization ON for an existing node (<c>UpdateTagAttributes</c>) grants the
/// HistoryWrite bit when an updater is wired — the same bit <c>EnsureVariable</c> sets at materialise — so
/// the newly historized node is updatable without a rebuild.
/// </summary>
[Fact]
public async Task Update_tag_attributes_turning_historization_on_sets_history_write()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
nm.HistorianDataUpdater = new FakeUpdater();
nm.EnsureVariable(RawPath, parentFolderNodeId: null, displayName: RawPath, dataType: "Float",
writable: false, historianTagname: null, realm: AddressSpaceRealm.Raw);
(nm.TryGetVariable(RawPath, AddressSpaceRealm.Raw)!.AccessLevel & AccessLevels.HistoryWrite).ShouldBe((byte)0);
nm.UpdateTagAttributes(RawPath, writable: false, historianTagname: Tagname, dataType: "Float",
isArray: false, arrayLength: null, realm: AddressSpaceRealm.Raw).ShouldBeTrue();
var access = nm.TryGetVariable(RawPath, AddressSpaceRealm.Raw)!.AccessLevel;
(access & AccessLevels.HistoryWrite).ShouldNotBe((byte)0);
(access & AccessLevels.HistoryRead).ShouldNotBe((byte)0);
}
finally { await host.DisposeAsync(); }
}
/// <summary>An updater that throws is <c>BadInternalError</c> for that node, never an exception out of the service.</summary>
[Fact]
public async Task An_updater_that_throws_is_bad_internal_error()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
nm.HistorianDataUpdater = new FakeUpdater { Throw = new InvalidOperationException("backend down") };
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var (results, errors) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(T1, 1d));
errors[0].StatusCode.Code.ShouldBe(StatusCodes.BadInternalError);
results[0].StatusCode.Code.ShouldBe(StatusCodes.BadInternalError);
}
finally { await host.DisposeAsync(); }
}
/// <summary>
/// DeleteRawModified needs both bounds (Part 11): a <c>DateTime.MinValue</c> ("unspecified") start or end is
/// <c>BadInvalidArgument</c> and never reaches the backend.
/// </summary>
[Fact]
public async Task Delete_raw_with_an_unspecified_bound_is_bad_invalid_argument()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var identity = KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate);
foreach (var (start, end) in new[] { (DateTime.MinValue, DateTime.MinValue), (DateTime.MinValue, T2), (T1, DateTime.MinValue) })
{
var (results, _) = Call(nm, identity, typeof(DeleteRawModifiedDetails),
new DeleteRawModifiedDetails { NodeId = nodeId, StartTime = start, EndTime = end, IsDeleteModified = false });
results[0].StatusCode.Code.ShouldBe(StatusCodes.BadInvalidArgument);
}
updater.DeleteRawCalls.ShouldBeEmpty();
}
finally { await host.DisposeAsync(); }
}
// ───────────────────────────── semantics ─────────────────────────────
/// <summary>
@@ -441,8 +546,9 @@ public sealed class HistoryUpdateTests : IDisposable
nm.HistoryUpdate(context, detailsType, nodesToUpdate, results, errors);
// Unadmitted (no HistoryWrite bit) slots are left untouched by the base and later filled by the
// master node manager; report them the way the client would see them.
// The base seeds every node THIS manager owns — with or without the HistoryWrite bit — with
// BadHistoryOperationUnsupported. A slot is only still null when GetManagerHandle found no node here
// (the master node manager would answer it); report that the way the client would see it.
if (results[0] is null) results[0] = new HistoryUpdateResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
errors[0] ??= new ServiceResult(StatusCodes.BadHistoryOperationUnsupported);
return (results, errors);
@@ -451,6 +557,9 @@ public sealed class HistoryUpdateTests : IDisposable
private static RoleCarryingUserIdentity KeyIdentity(params string[] roles) =>
new(new UserNameIdentityToken { UserName = "apikey" }, roles, Principal, KeyId);
private static RoleCarryingUserIdentity AnonymousIdentity(params string[] roles) =>
new(new AnonymousIdentityToken(), roles, OpcUaPrincipals.Anonymous, apiKeyId: null);
private static void WireAcl(
OtOpcUaNodeManager nm, bool enabled, NodePermissions flags, TimeProvider? clock = null,
DateTime? expiresUtc = null)
@@ -489,11 +598,13 @@ public sealed class HistoryUpdateTests : IDisposable
public uint[]? UpdateCodes { get; init; }
public uint DeleteRawCode { get; init; } = StatusCodes.Good;
public uint[]? DeleteAtTimeCodes { get; init; }
public Exception? Throw { get; init; }
public async Task<IReadOnlyList<uint>> UpdateDataAsync(
string tag, HistoryUpdateMode mode, IReadOnlyList<HistorizationValue> values, CancellationToken ct)
{
await Task.Delay(1, ct).ConfigureAwait(false);
if (Throw is not null) throw Throw;
lock (UpdateCalls) UpdateCalls.Add((tag, mode, values.ToList()));
return UpdateCodes ?? Enumerable.Repeat(StatusCodes.GoodEntryInserted, values.Count).ToArray();
}
@@ -218,6 +218,37 @@ public sealed class HistoryPagingTests
nextSkip.ShouldBe(0);
}
[Fact]
public void SliceTieCluster_reverse_drain_steps_down_one_tick_while_the_window_remains()
{
// Reverse read (start > end): a drained cluster at T resumes at T-1tick, not T+1tick (which lies outside
// a reverse window and used to end the read after its first page).
var t = new DateTime(2026, 1, 1, 0, 0, 5, DateTimeKind.Utc);
HistoryPaging.SliceTieCluster(
clusterCount: 1, skip: 1, cap: 1, boundaryT: t, endUtc: t.AddHours(-1),
out _, out var sliceCount, out var nextStartUtc, out var nextSkip, reverse: true);
sliceCount.ShouldBe(0);
nextStartUtc.ShouldBe(t.AddTicks(-1));
nextSkip.ShouldBe(0);
}
[Fact]
public void SliceTieCluster_reverse_drain_terminates_when_the_next_tick_is_the_excluded_end()
{
// The reverse window is (end, start]; resuming at exactly end would become a point read of the excluded
// sample, so T-1tick == end is terminal.
var t = new DateTime(2026, 1, 1, 0, 0, 5, DateTimeKind.Utc);
HistoryPaging.SliceTieCluster(
clusterCount: 2, skip: 0, cap: 5, boundaryT: t, endUtc: t.AddTicks(-1),
out _, out var sliceCount, out var nextStartUtc, out _, reverse: true);
sliceCount.ShouldBe(2);
nextStartUtc.ShouldBeNull();
}
[Fact]
public void SliceTieCluster_self_heals_when_skip_exceeds_cluster()
{
@@ -312,6 +312,70 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
await host.DisposeAsync();
}
/// <summary>
/// A reverse read (start &gt; end) with <c>NumValuesPerNode = 1</c> over three distinct-timestamp samples must
/// page through every sample, newest first. Before the fix the second page's resume read re-returned the
/// boundary sample, the trim emptied it, the tie-cluster path drained the one-sample "cluster" and stepped
/// FORWARD to <c>T + 1 tick</c> — past a reverse window's end — so paging stopped after the first sample.
/// </summary>
[Fact]
public async Task Raw_reverse_read_with_one_value_per_page_returns_every_sample_descending()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
nm.HistorianDataSource = new SeriesHistorianDataSource(MakeSeries(count: 3, stepSeconds: 1));
nm.EnsureVariable("eq-1/rev", parentFolderNodeId: null, displayName: "Rev", dataType: "Double",
writable: false, historianTagname: "WW.Rev", realm: AddressSpaceRealm.Uns);
var nodeId = nm.TryGetVariable("eq-1/rev")!.NodeId;
var collected = await DrainAsync(nm, nodeId, start: Epoch.AddHours(1), end: Epoch.AddSeconds(-1), max: 1);
collected.ShouldBe([2d, 1d, 0d]);
await host.DisposeAsync();
}
/// <summary>The reverse mirror of <see cref="Raw_oversized_tie_cluster_pages_within_the_timestamp"/>: a 5-way tie
/// cluster between distinct samples, cap 2, read newest-first — every value once, in descending order.</summary>
[Fact]
public async Task Raw_reverse_read_pages_through_an_oversized_tie_cluster()
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
nm.HistorianDataSource = new SeriesHistorianDataSource(
MakeSeriesWithTieCluster(distinctBefore: 3, tieCount: 5, distinctAfter: 2));
nm.EnsureVariable("eq-1/revburst", parentFolderNodeId: null, displayName: "RevBurst", dataType: "Double",
writable: false, historianTagname: "WW.RevBurst", realm: AddressSpaceRealm.Uns);
var nodeId = nm.TryGetVariable("eq-1/revburst")!.NodeId;
var collected = await DrainAsync(nm, nodeId, start: Epoch.AddHours(1), end: Epoch.AddSeconds(-1), max: 2);
collected.ShouldBe(Enumerable.Range(0, 10).Reverse().Select(i => (double)i));
await host.DisposeAsync();
}
/// <summary>Pages a Raw read to completion and returns the union of the pages in arrival order.</summary>
private static Task<List<double>> DrainAsync(OtOpcUaNodeManager nm, NodeId nodeId, DateTime start, DateTime end, uint max)
{
var collected = new List<double>();
byte[]? cp = null;
var pageCount = 0;
do
{
var (values, error, nextCp) = ReadRaw(nm, nodeId, start, end, max, inboundCp: cp);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
collected.AddRange(values);
cp = nextCp;
pageCount++;
pageCount.ShouldBeLessThan(40, "paging must terminate, not loop");
}
while (cp is not null);
return Task.FromResult(collected);
}
// --- helpers --------------------------------------------------------------------------------
/// <summary>Issue a single-node Raw HistoryRead and return that node's decoded sample values, error,
@@ -420,10 +484,18 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
CancellationToken cancellationToken)
{
Interlocked.Increment(ref _readCount);
var window = series
.Where(s => (s.SourceTimestampUtc ?? DateTime.MinValue) >= startUtc
&& (s.SourceTimestampUtc ?? DateTime.MinValue) <= endUtc)
.ToList();
// Reverse (Part 11, start > end): (end, start] in DESCENDING time — the SQL backend's contract.
// Ties keep series order reversed, so a reverse page of a cluster is deterministic.
var window = startUtc > endUtc
? series
.Where(s => (s.SourceTimestampUtc ?? DateTime.MinValue) > endUtc
&& (s.SourceTimestampUtc ?? DateTime.MinValue) <= startUtc)
.Reverse()
.ToList()
: series
.Where(s => (s.SourceTimestampUtc ?? DateTime.MinValue) >= startUtc
&& (s.SourceTimestampUtc ?? DateTime.MinValue) <= endUtc)
.ToList();
// maxValuesPerNode == 0 means "no limit".
var page = maxValuesPerNode == 0 ? window : window.Take((int)maxValuesPerNode).ToList();
return Task.FromResult(new HistorianRead(page, null));
@@ -0,0 +1,58 @@
using System.Diagnostics.Metrics;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Historian;
/// <summary>
/// The <c>otopcua.historian.outbox.unreadable</c> gauge reports the bound outbox's
/// <see cref="IHistorizationOutbox.UnreadableCount"/>. The listener is filtered by the instrument INSTANCE (the
/// shared <see cref="OtOpcUaTelemetry.Meter"/> object plus the instrument name), never by meter name alone, so a
/// parallel test's measurements on another meter of the same name cannot reach it; and the bound outbox reports a
/// distinctive count, so the asserted value can only have come from this test's outbox.
/// </summary>
public sealed class ContinuousHistorizationMetricsTests
{
private const string GaugeName = "otopcua.historian.outbox.unreadable";
[Fact]
public void The_unreadable_gauge_reports_the_bound_outboxs_unreadable_count()
{
const long distinctive = 7_331_919L;
ContinuousHistorizationMetrics.BindOutbox(new UnreadableOutbox(distinctive));
var observed = new List<long>();
using var listener = new MeterListener
{
InstrumentPublished = (instrument, l) =>
{
if (ReferenceEquals(instrument.Meter, OtOpcUaTelemetry.Meter) && instrument.Name == GaugeName)
l.EnableMeasurementEvents(instrument);
},
};
listener.SetMeasurementEventCallback<long>((instrument, value, _, _) =>
{
if (instrument.Name == GaugeName) observed.Add(value);
});
listener.Start();
listener.RecordObservableInstruments();
observed.ShouldBe([distinctive]);
}
private sealed class UnreadableOutbox(long unreadable) : IHistorizationOutbox
{
public long DroppedCount => 0;
public long UnreadableCount => unreadable;
public ValueTask AppendAsync(HistorizationOutboxEntry entry, CancellationToken ct) => ValueTask.CompletedTask;
public ValueTask AppendBatchAsync(IReadOnlyList<HistorizationOutboxEntry> entries, CancellationToken ct) => ValueTask.CompletedTask;
public ValueTask<IReadOnlyList<HistorizationOutboxEntry>> PeekBatchAsync(int max, CancellationToken ct) =>
ValueTask.FromResult<IReadOnlyList<HistorizationOutboxEntry>>([]);
public ValueTask RemoveAsync(Guid id, CancellationToken ct) => ValueTask.CompletedTask;
public ValueTask<int> CountAsync(CancellationToken ct) => ValueTask.FromResult(0);
public void Dispose() { }
}
}