Joseph Doherty
227 lines
31 KiB
Markdown
227 lines
31 KiB
Markdown
# Review 08 — Historian backends (Gateway / SqlServer / QuestDb / Sql.Common), Client.*, Analyzers
|
||
|
||
Repo: `/Users/dohertj2/Desktop/OtOpcUa` @ `d51d35bd` (master). Read-only review; every finding below was verified by reading the cited code. Line numbers are from the files as of this commit.
|
||
|
||
Severity: Critical / High / Medium / Low. Confidence: High = the code path is unambiguous; Medium = depends on runtime/SDK/gateway behaviour I could not execute here but which the repo's own comments or the package docs assert.
|
||
|
||
---
|
||
|
||
## 1. Memory / resource leaks
|
||
|
||
### 1.1 Uncapped raw HistoryRead (`NumValuesPerNode = 0`) materialises a tag's whole window in memory on every backend
|
||
- **Severity:** High **Confidence:** High
|
||
- **Location:**
|
||
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs:99-126`
|
||
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs:101-129`
|
||
- `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs:213-216, 285-305` and `Mapping/SampleMapper.cs:69-75`
|
||
- Node-manager side (outside slice, confirms there is no paging): `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs:116` ("`NumValuesPerNode == 0` … never page; return everything in one shot"); SqlServer/QuestDb report `MaxRawValuesPerRead => 0` (SqlServerHistorianDataSource.cs:324, QuestDbHistorianDataSource.cs:365).
|
||
- **Evidence:**
|
||
```csharp
|
||
// SqlServer
|
||
var top = maxValuesPerNode == 0 ? int.MaxValue : (int)Math.Min(maxValuesPerNode, int.MaxValue);
|
||
...
|
||
$"SELECT TOP (@top) {SampleColumns} FROM {_conn.Q("Sample")} WHERE TagId = @id AND {where} ORDER BY SourceTimeUtc {order}"
|
||
while (await reader.ReadAsync(...)) samples.Add(new DataValueSnapshot(...)); // List<DataValueSnapshot>, unbounded
|
||
// Gateway
|
||
if (cap == 0) { var all = await PassAsync(int.MaxValue) ... return all.Result.Samples; } // List<HistorianSample>
|
||
var snapshots = SampleMapper.ToSnapshots(samples); // second full copy
|
||
```
|
||
- **Failure scenario:** Any admitted session (reads are ungated for Anonymous with `Security:Acl:Enabled=false`, the default) issues HistoryRead-Raw with `NumValuesPerNode = 0` over a wide window on a dense tag (a year of 1 Hz = 31.5M rows). SqlServer streams the clustered-key seek into a `List<DataValueSnapshot>` (boxed value + two DateTimes + status ≈ 100+ B each ⇒ several GB), the Gateway path builds the protobuf list *and* the snapshot list, then the SDK encodes the whole response. `MaxTimeWeightedRawRows` caps only processed reads; `HistoryReadDeadline` (60 s) is time, not rows, and a key seek fills GBs in well under 60 s. The server's `TransportQuotas` are SDK defaults (`OpcUaApplicationHost.cs:472`), so the oversized response is rejected *after* the allocation. With `MaxConcurrentHistoryReadBatches = 16` this is a trivial memory-exhaustion vector, and it also works on the Gateway backend when the gateway is in streaming mode (the default `AllowStreamingBeyondCeiling`).
|
||
- **Suggested fix:** Treat `0` as "server-chosen page size" (Part 11 allows a server to return fewer values plus a continuation point): add a `ServerHistorian:MaxRawValuesPerRead`-style hard cap (advertise it as `HistoryServerCapabilities/MaxReturnDataValues`) and route `N = 0` through the existing `RawPageLimit` paging. Independently, bound the backends' `TOP/LIMIT`/`max_values` by that cap so a misbehaving caller cannot bypass the node manager.
|
||
|
||
### 1.2 Gateway events read drops `maxEvents` on the wire and abandons the gRPC stream at the cap
|
||
- **Severity:** High **Confidence:** Medium (the eviction behaviour is asserted by the repo's own comments and the gateway's `HistorianRpcFaults`, not re-verified here)
|
||
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs:111-124`; `GatewayHistorianDataSource.cs:576-598`; contrast `RawAggregateReader.cs:18-24` and `GatewayHistorianDataSource.cs:202-204`.
|
||
- **Evidence:**
|
||
```csharp
|
||
// Adapter: maxEvents is accepted and never forwarded (the 0.3.0 package API has no max parameter)
|
||
public IAsyncEnumerable<HistorianEvent> ReadEventsAsync(string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken ct)
|
||
{ ... return _inner.ReadEventsAsync(startUtc, endUtc, filter, ct); }
|
||
// Data source: breaks out of the server stream once the cap is full
|
||
if (hasCap && collected.Count == maxEvents) { truncated = true; break; }
|
||
```
|
||
while the raw path documents: "Every stream is consumed to completion (never broken out of — a cancelled gateway stream evicts its pooled historian session)" and `RawAggregateReader` says "the gateway currently classifies that cancellation as a connection fault and evicts its shared historian session (HistorianGateway `HistorianRpcFaults.IsConnectionFault`: any `RpcException` ⇒ true)".
|
||
- **Failure scenario:** Every capped HistoryReadEvents that has more than `NumValuesPerNode` events in its window (the normal case for a consumer paging alarm history, and for the #691.4 reverse walk-back which issues many small capped reads) cancels the server stream mid-way ⇒ the gateway evicts its historian session ⇒ the next read of *any* kind pays a reconnect, and concurrent raw reads sharing that session fail. Uncapped reads (`int.MaxValue`) additionally stream the entire window regardless of what the client asked for.
|
||
- **Suggested fix:** Drain the stream to completion with a bounded budget like the raw path (count past the cap without collecting), and add a `max_events` field to the gateway RPC so the gateway stops streaming at the cap; until then at least set `EventReadHorizon` guidance in docs.
|
||
|
||
### 1.3 `FasterLogHistorizationOutbox.PeekBatchAsync` holds a monitor across FasterLog disk reads, blocking the recorder actor's appends
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs:215-252` (peek) and `:147-156` (append index under the same lock); drain runs off-actor via `DrainOnceAsync(...).PipeTo` at `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:696`.
|
||
- **Evidence:**
|
||
```csharp
|
||
lock (_state)
|
||
{
|
||
...
|
||
using FasterLogScanIterator iter = _log.Scan(_nextScanAddress, _log.TailAddress, recover: false);
|
||
while (batch.Count < max && iter.GetNext(out byte[] bytes, out int len, out long address, out long next))
|
||
```
|
||
- **Failure scenario:** `FasterLog` is built with default settings (`new FasterLogSettings { LogDevice = _device }`, line 76), i.e. an ~8 MB in-memory window; `Capacity` defaults to 500 000 entries (≈35 MB on disk). Once the backlog's head has aged out of memory (any multi-minute gateway outage), `GetNext` reads pages from the device synchronously while `_state` is held; the actor thread's `AppendBatchAsync` then blocks on `lock (_state)` for the duration of the read — a sync block on an Akka dispatcher thread, once per 2 s drain pass, exactly during outage recovery.
|
||
- **Suggested fix:** Scan outside the lock (snapshot `_nextScanAddress` + the live-list cursor under the lock, iterate unlocked, re-take the lock only to flag/drop unreadable head records), or size `MemorySizeBits` to the configured capacity.
|
||
|
||
### 1.4 Client raw history read leaves the server continuation point unreleased when stopped by the client cap
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs:175-206` vs the modified-read path `:263-272`.
|
||
- **Evidence:** `while (continuationPoint != null && continuationPoint.Length > 0 && allValues.Count < maxValues);` then `return allValues;` — no `releaseContinuationPoints` call; `HistoryReadModifiedAsync` explicitly releases ("A read stopped by the client's own cap leaves a server cursor behind; release it.").
|
||
- **Failure scenario:** A page that returns exactly `maxValues` (the common case on a dense tag) carries a continuation point the client never resumes or releases. Bounded in practice: the server's store is per-session and capped at `MaxHistoryContinuationPoints` (100, oldest-evicted, `OpcUaServer/HistoryContinuationStore.cs:50-55`), so the Client.UI's long-lived session just cycles 100 slots; the CLI disconnects immediately.
|
||
- **Suggested fix:** Mirror the modified path's release call.
|
||
|
||
### 1.5 Client subscription adapters delete server-side but never remove the `Subscription` from the SDK `Session`
|
||
- **Severity:** Low **Confidence:** Medium
|
||
- **Location:** `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs:321-333` (`_session.AddSubscription(subscription)`), `DefaultSubscriptionAdapter.cs:104-117, 122-133` (`_subscription.DeleteAsync(true, ct)` only).
|
||
- **Failure scenario:** `OpcUaClientService.SubscribeAlarmsAsync/UnsubscribeAlarmsAsync` and `MainWindowViewModel.MonitorAlarmsForSelectedNodeAsync` create a fresh SDK `Subscription` per cycle; each is deleted on the server but stays in `Session.Subscriptions` until the session is disposed, so a UI session that re-targets alarms repeatedly accumulates dead `Subscription` objects (and anything the SDK does over `Session.Subscriptions`, e.g. transfer/recreate on its own reconnect, iterates them).
|
||
- **Suggested fix:** Call `_session.RemoveSubscription(subscription)` after a successful delete (the adapter would need the session reference), or create subscriptions through `Session.RemoveSubscription`-aware helpers.
|
||
|
||
---
|
||
|
||
## 2. Unimplemented / stubbed / silently wrong
|
||
|
||
### 2.1 Gateway native `TimeAverage` / `Total` are served with **Good** status although the repo documents the values are wrong until `historiangw#27`
|
||
- **Severity:** Medium **Confidence:** High
|
||
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/NativeAggregatePlan.cs:39-43`; served via `GatewayHistorianDataSource.cs:338-343, 350-387`; advertised in `ServedAggregates` (`:40-43`).
|
||
- **Evidence:**
|
||
```
|
||
⚠️ Today historiangw#27 (the interval reaches AVEVA in ticks where it reads milliseconds) makes AVEVA answer ONE
|
||
Good row per request — the whole-intervals read lands on its earliest interval with a whole-span value and the rest
|
||
are absent — in both directions.
|
||
```
|
||
(`.claude/rules/historian.md` agrees: "until historiangw#27 lands both directions return Good-status wrong values".)
|
||
- **Failure scenario:** A client asks for 24 hourly `TimeAverage` intervals and receives one Good, whole-day value stamped on the first interval and 23 absent intervals; nothing on the wire says the answer is degraded. ScadaBridge or a report will store it as truth.
|
||
- **Suggested fix:** Until the gateway fix ships, either drop `TimeAverage`/`Total` from `ServedAggregates` (⇒ `BadAggregateNotSupported`, consistent with the other five time-weighted aggregates on this backend) or mark every row `UncertainDataSubNormal` + log once. Gate the native path behind a config flag that defaults off until #27 is live-verified.
|
||
|
||
### 2.2 Gateway backend capability gaps vs. SqlServer/QuestDb (documented, but a consumer-visible matrix)
|
||
- **Severity:** Medium **Confidence:** High
|
||
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianProvider.cs:54` (`CreateUpdater => null`), `GatewayHistorianDataSource.cs:40-43` (12 of 17 aggregates), `:125-126` and `:556-562` (reverse raw/events `NotSupportedException`), `ReadModifiedAsync` not overridden (interface default ⇒ unsupported), no `CreateRetention`, `GatewayHistorianValueWriter.cs:28-38` (quality discarded by the gateway — "C-002"), `Mapping/HistorianTypeMapper.cs` (String/DateTime not provisioned).
|
||
- **What a client sees on Gateway only:** HistoryUpdate ⇒ `BadHistoryOperationUnsupported`; `IsReadModified` ⇒ unsupported; `TimeAverage2/Total2/Interpolative/PercentGood/PercentBad` ⇒ `BadAggregateNotSupported`; every stored sample reads back Good (comms-loss markers are dropped, driver Bad statuses are lost); reverse/end-only raw and events reads are emulated by forward walks (`ReverseRawReader`/`ReverseEventsReader`) that cost many round trips on sparse tags.
|
||
- **Suggested fix:** Nothing to "implement" in this repo for most of it (gateway-side), but publish the matrix in `docs/Historian.md` and have the AdminUI backend picker show it; the quality loss in particular deserves a startup Warning when `ContinuousHistorization` is enabled on Gateway.
|
||
|
||
### 2.3 Client CLI/UI offer `StandardDeviation`, which no backend serves, and omit six aggregates that are served
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs` (12 members incl. `StandardDeviation`); `Client.CLI/Commands/HistoryReadCommand.cs:50, 202-221`; `Client.UI/ViewModels/HistoryViewModel.cs:47-62`. Served set: `Sql.Common/AggregateFamily.cs:50-51` (`FamilyOf != null` ⇒ 17: Average/Min/Max/Count/Range/Start/End/Delta/Min-MaxActualTime/TimeAverage/TimeAverage2/Total/Total2/Interpolative/PercentGood/PercentBad).
|
||
- **Failure scenario:** `historyread --aggregate stddev` always returns `BadAggregateNotSupported`; `Range`, `Delta`, `MinimumActualTime`, `MaximumActualTime`, `PercentGood`, `PercentBad` cannot be requested from the project's own client at all, so the gate recipes in `docs/Historian.md` cannot be run with Client.CLI.
|
||
- **Suggested fix:** Align `AggregateType` + `AggregateTypeMapper` with `AggregateFamilies.Served`.
|
||
|
||
### 2.4 `ProbeAsync` / `GetConnectionStatusAsync` exist on the gateway seam but health is derived from read outcomes only
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `IHistorianGatewayClient.cs:106, 111`; `GatewayHistorianDataSource.cs:616-638` (`ProcessConnectionOpen: _totalSuccesses > 0 && _consecutiveFailures == 0`).
|
||
- **Failure scenario:** A freshly booted node with a healthy gateway reports the historian as closed until some client performs a HistoryRead; a node that only *writes* (continuous historization) never flips it. Operators reading `/hosts` get a false negative.
|
||
- **Suggested fix:** Use `ProbeAsync` on a slow timer (or on `GetHealthSnapshot` with a cached result) now that the seam carries it.
|
||
|
||
### 2.5 Two Gateway read arms record a caller cancellation as a backend failure
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `GatewayHistorianDataSource.cs:607-612` (`ReadEventsAsync`) and `:519-524` (`ReadAtTimeNativeAsync`) lack the `catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) throw;` that `ReadRawAsync`/`ReadProcessedAsync`/`ReadAtTimeAsync` have (`:141-144, 377-380, 484-487`).
|
||
- **Failure scenario:** A `HistoryReadDeadline` expiry or client abort during an events read bumps `TotalFailures`/`ConsecutiveFailures` and flips `ProcessConnectionOpen` to false ⇒ health alarms on a healthy gateway.
|
||
|
||
---
|
||
|
||
## 3. Performance limiters
|
||
|
||
### 3.1 QuestDB live-write path: three round trips plus a 10 ms-poll visibility wait per tag group per drain pass
|
||
- **Severity:** Medium **Confidence:** High
|
||
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs:91-102, 128-143`; `QuestDbWalVisibility.cs:49-91`.
|
||
- **Evidence:**
|
||
```csharp
|
||
await _catalog.EnsureAsync(tag, dataType, ct); // cached after first
|
||
var present = await ExistingTicksAsync(tag, rows, ct); // SELECT ts ... per cluster
|
||
for (...) await InsertAsync(...); // multi-row INSERT
|
||
await _wal.WaitAsync(_conn.SampleTable, ct); // SELECT sequencerTxn, then poll writerTxn every 10 ms ≤ 5 s
|
||
```
|
||
- **Failure scenario:** With `DrainConcurrency = 8` and WAL apply lagging under load, up to eight pollers each issue ~100 `wal_tables()` queries/s (≈800/s) against the same database that is already behind — the wait amplifies load exactly when the DB is slowest, and every tag group pays ≥3 round trips, so a 5 000-tag deployment at 1 s cadence spends `⌈5000/8⌉ × (3 RTT + apply latency)` per pass and falls behind the 2 s cadence. The existence pre-read is only needed for first-wins idempotency across redeliveries, and the visibility wait only so that an *immediate* redelivery sees the rows.
|
||
- **Suggested fix:** Wait once per drain pass (expose a `FlushAsync`/pass-end hook on `IHistorianValueWriter`, or let the recorder call `WaitAsync` once after `Task.WhenAll`), back the poll off exponentially (10→50→200 ms), and consider a per-pass existence read keyed by the outbox batch rather than per tag.
|
||
|
||
### 3.2 All three alarm-history writers insert one event per round trip, serially
|
||
- **Severity:** Medium **Confidence:** High
|
||
- **Location:** `Gateway/GatewayAlarmHistorianWriter.cs:66-78` (one `SendEventAsync` per event); `SqlServer/SqlServerAlarmHistorianWriter.cs:55-59, 76-89` (new pooled connection + single-row `INSERT … WHERE NOT EXISTS` per event); `QuestDb/QuestDbAlarmHistorianWriter.cs:64-69, 89-105`.
|
||
- **Failure scenario:** The `LocalDbStoreAndForwardSink` drains `BatchSize` events per tick; an alarm storm of 10 000 events drains at `10 000 × RTT` (≥ 10–20 s on SQL, minutes over a WAN to the gateway), and the Primary-only drain means the backlog sits in LocalDb meanwhile. The per-event design exists to isolate poison events, but SqlServer already has the TVP + `OUTPUT` machinery to get per-row outcomes from one statement.
|
||
- **Suggested fix:** SqlServer: one `INSERT … SELECT FROM @events WHERE NOT EXISTS …` with `OUTPUT inserted.EventId` and map outcomes by id (fall back to per-row only on a data error). QuestDb: multi-row INSERT (DEDUP key already converges duplicates). Gateway: needs a batch `SendEvents` RPC.
|
||
|
||
### 3.3 SQL Server retention issues O(tags) `DELETE TOP` statements per purge even when nothing is eligible
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs:89-107, 134-149`.
|
||
- **Evidence:** `SELECT TagId FROM Tag` then `foreach (var id in tagIds) { DeleteLoopAsync(Sample…); if (trail) DeleteLoopAsync(SampleModified…); }` — the loop always runs at least one statement per table per tag, and the applock is held for the whole pass.
|
||
- **Failure scenario:** 50 000 historized tags ⇒ ≥100 000 statements per hourly purge (plus the full `Event` scan) on a database that has nothing to delete, serialised on one connection while other nodes' purges skip.
|
||
- **Suggested fix:** Pre-filter: `SELECT TagId FROM Tag t WHERE EXISTS (SELECT 1 FROM Sample s WHERE s.TagId = t.TagId AND s.SourceTimeUtc < @cutoff)` (a key seek per tag inside one statement), or keep a per-tag `OldestSampleUtc` high-water mark.
|
||
|
||
### 3.4 QuestDB `event.source` and `sample_modified.tag` symbols carry no index
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `QuestDb/QuestDbTables.cs:73-77` (`source SYMBOL`), `:95-99` (`tag SYMBOL`), vs the sample table's `tag SYMBOL INDEX` added for #683 (`:64-67`, `:116`).
|
||
- **Failure scenario:** `ReadEventsAsync` with a `sourceName` filter (`QuestDbHistorianDataSource.cs:402-407`) and every `ReadModifiedAsync` (`:183-185`) scan all partitions in the window rather than seeking; the #683 gate measured the same shape costing seconds on the sample table.
|
||
- **Suggested fix:** Add `INDEX` on both symbols in the DDL plus the same idempotent `ALTER … ADD INDEX` migration the bootstrapper already runs for `sample.tag`.
|
||
|
||
### 3.5 `GatewayTagPresenceCache` adds a catalog-browse RPC before HistoryReads, keyed by a glob filter
|
||
- **Severity:** Low **Confidence:** Medium
|
||
- **Location:** `Gateway/GatewayTagPresenceCache.cs:16-32` (60 s positive / 5 s negative TTL, 4 096 entries); `HistorianGatewayClientAdapter.cs:85-93` (`BrowseTagNamesAsync(tag, ct)` drained fully). Package doc (0.3.0 XML): `BrowseTagNamesAsync(filter)` — "Tag name filter pattern accepted by the historian (e.g. `Sys*`)".
|
||
- **Failure scenario:** A consumer sweeping history for N tags pays N extra streaming RPCs per minute of sweep; with more than 4 096 distinct tags the LRU-less eviction (`_entries.Keys.First()`) thrashes. A historian tag name containing `*`/`?` (legal in AVEVA) streams every matching catalog entry on each check.
|
||
- **Suggested fix:** Use the package's `GetTagMetadataAsync` (exact lookup) now that the seam exists, lengthen the positive TTL, or drop the pre-check and treat an empty raw read as "no data" (which it already does).
|
||
|
||
### 3.6 Client: N+1 browse in `BrowseAsync`, and O(n) scan per notification in the UI
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `Client.Shared/OpcUaClientService.cs:223-242` (`HasChildrenAsync` ⇒ one Browse per Object child); `Client.UI/ViewModels/SubscriptionsViewModel.cs:61-73` (`foreach (var item in ActiveSubscriptions) if (item.NodeId == e.NodeId)` on the UI thread per notification).
|
||
- **Failure scenario:** `subscribe --recursive` / the UI's recursive subscribe over a 50 000-node raw tree issues ~50 000 Browse calls to find children; 2 000 subscribed items at 1 Hz = 4 M string compares/s on the Avalonia UI thread.
|
||
- **Suggested fix:** Browse with `includeSubtypes`/`resultMask` carrying `NodeClass` only and infer `HasChildren` lazily on expand; index `ActiveSubscriptions` by NodeId in a dictionary.
|
||
|
||
---
|
||
|
||
## 4. Other hazards
|
||
|
||
### 4.1 Every production `IHistorianValueWriter` swallows `OperationCanceledException`, so the recorder's shutdown guard is dead code
|
||
- **Severity:** Medium **Confidence:** High
|
||
- **Location:** `Gateway/Recorder/GatewayHistorianValueWriter.cs:112-120`; `SqlServer/SqlServerHistorianValueWriter.cs:93-97`; `QuestDb/QuestDbHistorianValueWriter.cs:104-108` — all `catch (Exception) { …; return false; }`. The consumer expects the opposite: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:744-755`:
|
||
```csharp
|
||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||
{
|
||
// Shutdown cut this write short. ... swallowed as `ok = false` it becomes a consecutive failure, and when a
|
||
// sibling group already wrote this pass the fold books it — so a routine node stop can tip a perfectly
|
||
// HEALTHY tag over its attempt budget and discard its queued samples on the way out. Rethrow ...
|
||
throw;
|
||
}
|
||
```
|
||
- **Failure scenario:** At shutdown mid-pass the cancelled groups return `false`; if any sibling already wrote, `_tagWriteFailures[tag]++` for each cancelled tag (recorder `:826-834`) — precisely the outcome the guard was written to prevent. Today the damage is bounded (+1 per process stop, counters are in-memory, `_lifetime` cancels only in `PostStop`, recorder `:340`), so no data is lost, but the contract between writer ("never throws") and recorder ("rethrow cancellation") is contradictory and the next refactor that cancels drains without restarting (a capture-gate close, a backend switch) will discard healthy tags' backlog after `MaxWriteAttempts` such events.
|
||
- **Suggested fix:** `catch (OperationCanceledException) when (ct.IsCancellationRequested) { throw; }` ahead of the catch-all in all three writers (the alarm writers already special-case it), and state it on `IHistorianValueWriter`.
|
||
|
||
### 4.2 Gateway value writer relabels a Local-kind timestamp as UTC instead of converting it
|
||
- **Severity:** Medium **Confidence:** Medium (depends on a driver ever producing `DateTimeKind.Local`)
|
||
- **Location:** `Gateway/Recorder/GatewayHistorianValueWriter.cs:93-98`:
|
||
```csharp
|
||
live.Timestamp = Timestamp.FromDateTime(DateTime.SpecifyKind(timestampUtc, DateTimeKind.Utc));
|
||
```
|
||
vs the SQL backends' shared normaliser `Sql.Common/LiveWritePreparer.cs:31`: `t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc);`.
|
||
- **Failure scenario:** A driver that stamps `DateTime.Now`-derived source times (or an OpcUaClient upstream server returning Local) historizes correctly on SqlServer/QuestDb but shifted by the node's UTC offset on Gateway — and the HistoryUpdate path on SQL backends keeps the kind (`HistoryUpdatePlanner` ⇒ `LiveWritePreparer.Truncate` only), so the same value can land at two different times on two backends.
|
||
- **Suggested fix:** Run gateway writes through `LiveWritePreparer.Prepare` (it already does the non-finite ⇒ Empty and dedup work the gateway path lacks), or at minimum `ToUniversalTime()` for `Local`.
|
||
|
||
### 4.3 SqlServer `DeleteRaw` removes an unbounded range in a single statement/transaction with `OUTPUT INTO`
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `SqlServer/SqlServerHistorianDataUpdater.cs:99-113`.
|
||
- **Failure scenario:** `DELETE FROM Sample OUTPUT … INTO SampleModified WHERE TagId=@id AND SourceTimeUtc >= @s AND < @e` over years of a 1 Hz tag deletes tens of millions of rows and writes as many audit rows in one transaction: lock escalation on `Sample` (blocking the live writer), transaction-log growth, and a 30 s `CommandTimeout` that aborts and rolls back the whole thing — the retention service deliberately batches `DELETE TOP (n)` for this reason. QuestDB's updater already pages (`QuestDbHistorianDataUpdater.cs:135-166`).
|
||
- **Suggested fix:** Page the delete (`DELETE TOP (@n) … OUTPUT …` in a loop under one transaction or per-batch commits; the OPC UA result is a single status either way).
|
||
|
||
### 4.4 QuestDB HistoryUpdate: non-atomic trail + per-instance locks + duplicating retries (documented residuals)
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `QuestDb/QuestDbHistorianDataUpdater.cs:18-21, 28-34, 155-160, 235-240`; `.claude/rules/historian.md` ("a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key)").
|
||
- **Failure scenario:** A crash or failed sample INSERT after `WriteTrailAsync` leaves orphan `sample_modified` rows describing changes that never landed; a second node (or the live writer) racing the same key bypasses the 64 striped `SemaphoreSlim`s; a Npgsql timeout retry duplicates trail rows. All documented; listed so the matrix is complete.
|
||
- **Suggested fix:** Record a `applied` marker (a second append with the same `modified_id`) after the sample write so `ReadModifiedAsync` can drop orphans; use a DEDUP key `(ts, tag, modified_id)` on the trail.
|
||
|
||
### 4.5 Client: failover nulls subscription state outside the lock the rest of the class relies on
|
||
- **Severity:** Low **Confidence:** High
|
||
- **Location:** `Client.Shared/OpcUaClientService.cs:734-735` (`_dataSubscription = null; _alarmSubscription = null;` in `RunFailoverAsync`, no lock) vs `:260-263` (`SubscribeAsync` reads/creates `_dataSubscription` unguarded) and the class comment `:18-22` that promises all bookkeeping is under `_subscriptionLock`.
|
||
- **Failure scenario:** A `SubscribeAsync` racing a keep-alive failover can add a monitored item to the dying session's subscription (silently lost, not replayed) or observe a null and create a second subscription on the new session that `ReplaySubscriptionsAsync` then duplicates.
|
||
- **Suggested fix:** Take `_subscriptionLock` around the null-outs and guard `SubscribeAsync` with the same snapshot pattern `DisconnectAsync` uses; also `ConnectAsync` (`:97-128`) should refuse (or disconnect) when already connected instead of overwriting `_session`.
|
||
|
||
---
|
||
|
||
## 5. Checked and fine (looked like problems, are not)
|
||
|
||
- **SQL injection:** every dynamic identifier is regex-validated before splicing — SqlServer `Schema` `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (`SqlServerHistorianOptions.cs:49-59`, `SqlHistorianConnection.Q` brackets it, `$(Schema)` substituted only after validation), QuestDb `TablePrefix` `^[A-Za-z0-9_]+$` (`QuestDbHistorianOptions.cs:32-38`, re-checked in `QuestDbTables.Name`), `table_columns('{table}')` uses that validated name, QuestDB `AtTimeSql` inlines only generated ints, TTL days is `int`. All values (tag names, times, principals, event fields) are parameters; the SqlServer alarm writer deliberately uses `NVARCHAR(MAX)` params so truncation is a server error, not a silent clip.
|
||
- **Continuous-historization buffer bound:** the FasterLog outbox is disk-backed with `Capacity` (500 000 default) drop-oldest and `DroppedCount`/`UnreadableCount` metrics; drop-oldest truncates + commits, corrupt head records cannot pin the head (`FasterLogHistorizationOutbox.cs:172-200, 243-250`). Not unbounded.
|
||
- **Flush on shutdown:** `PerEntry` commit mode fsyncs before `AppendBatchAsync` returns; `Dispose` cancels the periodic loop, awaits it, then `Commit(spinWait: true)` (`:365-406`). Appends in flight at process kill are the documented `Periodic` trade-off only.
|
||
- **Time-weighted aggregates stream, they do not materialise:** both SQL backends feed `TimeWeightedAggregator` row by row with `TOP/LIMIT (cap + 1)` and `CheckWindowRows` fails the node one row past `MaxTimeWeightedRawRows`; memory is O(buckets) (`SqlServerHistorianDataSource.cs:528-550`, `QuestDbHistorianDataSource.cs:545-579`, `TimeWeightedAggregator.cs:132-154`). Gateway per-sample aggregates likewise accumulate into `RawBucketAccumulator` (O(buckets)).
|
||
- **`BucketPlan.MaxBuckets = 100 000`** bounds every processed read's result list and the aggregator arrays before any I/O.
|
||
- **Tag catalogs** (`SqlServerTagCatalog._cache`, `QuestDbTagCatalog._known`) are positive-only and never pruned, but are bounded by the number of distinct historian tag names ever written on this node (≤ a few hundred KB); a renamed tag leaves one stale int entry. Not a leak worth fixing.
|
||
- **Gateway channels:** four `HistorianGatewayClientAdapter`s (data source, alarm writer, provisioner, value writer) each own a `GrpcChannel`, each disposed by its owner (`DisposeAsync` on all four). Documented as deliberate; a co-located sidecar tolerates four HTTP/2 connections.
|
||
- **`GatewayRowCeiling`** is a single `int` + ticks with `Volatile`/`Interlocked`; a race costs one extra refusal, never a wrong answer. `GatewayAtTimeReader` is capped at `64 + 16 × times` reads and `1 024` rows per read, so a pathological tag fails `BadTooManyOperations` rather than looping.
|
||
- **Schema bootstrap:** `SqlServerSchemaBootstrapper`/`QuestDbSchemaBootstrapper` use `SemaphoreSlim(1,1)` + a 5 s negative cache (`BootstrapFailureCache`), so an outage does not serialise every call behind one round trip; the SqlServer applock is session-owned and the pool is cleared when release fails (otherwise the lock would ride a pooled connection forever) — same pattern in retention.
|
||
- **Timestamps on the SQL backends:** `datetime2(7)` with explicit UTC `SpecifyKind` on read/write; QuestDB rounds to µs and nudges `Min/MaxValue` so Npgsql never binds ±infinity; pre-1970 is refused/dropped with counters rather than failing a batch. `IHistorianDataSource.TimestampResolutionTicks = 10` lets the paging step correctly on QuestDB.
|
||
- **`GatewayHistorianDataSource.Dispose()`** is sync-over-async (`DisposeAsync().AsTask().GetAwaiter().GetResult()`) — shutdown-only, channel shutdown is quick; acceptable.
|
||
- **Analyzer (`UnwrappedCapabilityCallAnalyzer`)**: registers nothing when the guarded interfaces are absent from the compilation, resolves symbols once per compilation, and walks ancestors only up to the enclosing wrapper invocation; the one acknowledged limitation (any lambda argument to `ExecuteAsync` suppresses) is documented and currently unreachable. No production hazard.
|
||
- **Client `DisposeBridge`** bounds every sync-over-async teardown (`CloseAsync`/`DeleteAsync`) at 5 s and cancels the token when the wait loses, so a dead server cannot hang `Dispose`.
|
||
- **`HistorizationOutboxEntrySerializer`** is bounds-checked for both layouts (negative length prefixes, short records, invalid DateTime binary, non-0/1 booleans all return `false`), so a corrupt record is counted and skipped, never thrown.
|