dohertj2/lmxopcua
1
0
Fork 0
Code Issues 54 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(historian): backend capability matrix; warn on Gateway continuous historization (lmxopcua#746)

Browse Source 
The Gateway backend's gaps against SqlServer/QuestDb were documented
piecemeal. The backend-selection table in docs/Historian.md now also covers
the aggregate set (and the native time-weighted opt-in), emulated reverse
and end-only reads, the client-side events cap, retention and the health
probe.

Continuous historization on Gateway silently loses sample quality (stored
Good), comms-loss outage markers and non-numeric values. Building its live
writer, which only happens when ContinuousHistorization is enabled, now
logs one startup Warning that names the gaps and the alternatives.

The review asked for the matrix in the AdminUI backend picker, but no such
picker exists: the backend is configuration-only. Filed as a follow-up.

Rebased from the unpushed remed/lane-H2 branch (originally authored 2026-10-22
in session_01Ub9hGmYLQvn9XETzw7CMsT) onto current master.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-10-22 17:36:04 -04:00
parent 64269573e7
commit 096a3d5b3c
3 changed files with 78 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+10 -1
View File
@@ -77,13 +77,22 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` |
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) |
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
| Processed aggregates | Twelve: the ten per-sample ones (Average, Count, Minimum, Maximum, Range, Start, End, Delta, Min/MaxActualTime) evaluated from raw reads with the SQL backends' evaluator — identical results (#681) — plus TimeAverage and Total from AVEVA's native modes | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results |
| Processed aggregates | **Ten** by default: the per-sample ones (Average, Count, Minimum, Maximum, Range, Start, End, Delta, Min/MaxActualTime) evaluated from raw reads with the SQL backends' evaluator — identical results (#681). TimeAverage and Total from AVEVA's native modes **only with `ServerHistorian:Gateway:EnableNativeTimeWeighted=true`** (default off: Good-status wrong values until historiangw#27). TimeAverage2 / Total2 / Interpolative / PercentGood / PercentBad: never | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results |
| Reverse / end-only Raw and Events reads | Emulated: walked back from bounded forward reads by the server (`ReverseRawReader` / `ReverseEventsReader`) — correct, but many round trips on a sparse tag | Native newest-first | Native newest-first |
| Capped Events read | The cap is applied client-side (the gateway RPC has no max-events field); the stream past it is drained up to `Gateway:EventDrainBudget` rows | `TOP` | `LIMIT` |
| Retention | AVEVA Historian's own | `SqlServer:RetentionDays` purge (opt-in) | `QuestDb:RetentionDays` TTL (opt-in) |
| Health connection flags | `Gateway:ProbeInterval` probe + read outcomes | Read outcomes | Read outcomes |
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change, marked applied after it (an orphan is not served) |
| Alarm history | gateway `SendEvent`, one RPC per event, up to 8 in flight (no batch RPC yet) | `Event` table (insert-if-absent), multi-row statements of 150 events | `{prefix}event` table (dedup upsert on `(ts, event_id)`), multi-row INSERTs of 500 events |
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
**Continuous historization on `Gateway` logs a startup Warning** (when its live writer is built, i.e. only with
`ContinuousHistorization:Enabled`): every sample's quality is discarded (stored Good), comms-loss outage markers are dropped,
and non-numeric values are not recorded — the rows above. The AdminUI has no backend picker (the backend is configuration-only);
this table is the place to compare them.
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend;
HistoryRead is served from any node (the database is shared). HistoryUpdate is **Primary-only** since #674
(`BadInvalidState` on any other node — see [HistoryUpdate](#primary-only-674)). The historian connection string is
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianProvider.cs
+11
View File
@@ -35,15 +35,26 @@ public sealed class GatewayHistorianProvider : IHistorianBackendProvider
public IHistorianDataSource CreateDataSource(IServiceProvider sp) => GatewayHistorian.CreateDataSource(_options, sp);
/// <inheritdoc />
// Built only when continuous historization is enabled on this backend, so this is where its startup Warning lives:
// the gaps are silent at run time (the gateway accepts a quality it then discards).
public IHistorianValueWriter CreateValueWriter(IServiceProvider sp)
{
ArgumentNullException.ThrowIfNull(sp);
var loggerFactory = sp.GetService<ILoggerFactory>() ?? NullLoggerFactory.Instance;
var logger = sp.GetService<ILogger<GatewayHistorianValueWriter>>()
?? NullLogger<GatewayHistorianValueWriter>.Instance;
logger.LogWarning(ContinuousHistorizationWarning);
return new GatewayHistorianValueWriter(HistorianGatewayClientAdapter.Create(_options, loggerFactory), logger);
}
/// <summary>The startup Warning for continuous historization on this backend (see docs/Historian.md, the capability matrix).</summary>
internal const string ContinuousHistorizationWarning =
"Continuous historization is enabled on the Gateway historian backend, which stores less than the SQL backends: " +
"the gateway discards each sample's quality (AVEVA stores every recorded value Good, so a driver's Bad or Uncertain " +
"publish is historized as Good), comms-loss outage markers are dropped (reads interpolate across an outage), and " +
"non-numeric values (String, DateTime) are not recorded. Use the SqlServer or QuestDb backend where stored status, " +
"outage markers or non-numeric history matter.";
/// <inheritdoc />
public IAlarmHistorianWriter CreateAlarmWriter(IServiceProvider sp) => GatewayHistorian.CreateAlarmWriter(_options, sp);
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayContinuousHistorizationWarningTests.cs
+57
View File
@@ -0,0 +1,57 @@
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Recorder;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// Continuous historization on the Gateway backend silently loses what the SQL backends keep — every sample's
/// quality (the gateway stores Good), comms-loss outage markers and non-numeric values — so building the live writer
/// (which only happens when continuous historization is enabled) logs one startup Warning naming the gaps.
/// </summary>
public sealed class GatewayContinuousHistorizationWarningTests
{
private sealed class CapturingProvider : ILoggerProvider
{
public readonly List<(LogLevel Level, string Message)> Entries = new();
public ILogger CreateLogger(string categoryName) => new L(this);
public void Dispose() { }
private sealed class L(CapturingProvider owner) : ILogger
{
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter)
{
lock (owner.Entries) owner.Entries.Add((logLevel, formatter(state, exception)));
}
}
}
[Fact]
public async Task Building_the_gateway_live_writer_warns_once_about_what_the_backend_drops()
{
var capture = new CapturingProvider();
using var sp = new ServiceCollection().AddLogging(b => b.AddProvider(capture)).BuildServiceProvider();
var provider = new GatewayHistorianProvider(
new ServerHistorianOptions { Endpoint = "http://localhost:5222", UseTls = false, ApiKey = "histgw_test_secret" });
var writer = Assert.IsType<GatewayHistorianValueWriter>(provider.CreateValueWriter(sp));
var warning = capture.Entries.Where(e => e.Level == LogLevel.Warning).ShouldHaveSingleItem().Message;
warning.ShouldContain("quality");
warning.ShouldContain("Good");
warning.ShouldContain("outage");
warning.ShouldContain("non-numeric");
warning.ShouldContain("SqlServer");
await writer.DisposeAsync();
// Positive control in the other direction: the read path alone (no continuous historization) does not warn.
capture.Entries.Clear();
await using var ds = (GatewayHistorianDataSource)provider.CreateDataSource(sp);
capture.Entries.ShouldNotContain(e => e.Level == LogLevel.Warning);
}
}