lmxopcua/docs/reviews/2026-10-21/08-historians-clients.md

Review 08 — Historian backends (Gateway / SqlServer / QuestDb / Sql.Common), Client.*, Analyzers

Repo: /Users/dohertj2/Desktop/OtOpcUa @ d51d35bd (master). Read-only review; every finding below was verified by reading the cited code. Line numbers are from the files as of this commit.

Severity: Critical / High / Medium / Low. Confidence: High = the code path is unambiguous; Medium = depends on runtime/SDK/gateway behaviour I could not execute here but which the repo's own comments or the package docs assert.

---

1. Memory / resource leaks

1.1 Uncapped raw HistoryRead (NumValuesPerNode = 0) materialises a tag's whole window in memory on every backend

• Severity: High Confidence: High
• Location:
  • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs:99-126
  • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs:101-129
  • src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs:213-216, 285-305 and Mapping/SampleMapper.cs:69-75
  • Node-manager side (outside slice, confirms there is no paging): src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs:116 ("NumValuesPerNode == 0 … never page; return everything in one shot"); SqlServer/QuestDb report MaxRawValuesPerRead => 0 (SqlServerHistorianDataSource.cs:324, QuestDbHistorianDataSource.cs:365).
• Evidence:

  // SqlServer
  var top = maxValuesPerNode == 0 ? int.MaxValue : (int)Math.Min(maxValuesPerNode, int.MaxValue);
  ...
  $"SELECT TOP (@top) {SampleColumns} FROM {_conn.Q("Sample")} WHERE TagId = @id AND {where} ORDER BY SourceTimeUtc {order}"
  while (await reader.ReadAsync(...)) samples.Add(new DataValueSnapshot(...));   // List<DataValueSnapshot>, unbounded
  // Gateway
  if (cap == 0) { var all = await PassAsync(int.MaxValue) ... return all.Result.Samples; }   // List<HistorianSample>
  var snapshots = SampleMapper.ToSnapshots(samples);                                           // second full copy
  

• Failure scenario: Any admitted session (reads are ungated for Anonymous with Security:Acl:Enabled=false, the default) issues HistoryRead-Raw with NumValuesPerNode = 0 over a wide window on a dense tag (a year of 1 Hz = 31.5M rows). SqlServer streams the clustered-key seek into a List<DataValueSnapshot> (boxed value + two DateTimes + status ≈ 100+ B each ⇒ several GB), the Gateway path builds the protobuf list and the snapshot list, then the SDK encodes the whole response. MaxTimeWeightedRawRows caps only processed reads; HistoryReadDeadline (60 s) is time, not rows, and a key seek fills GBs in well under 60 s. The server's TransportQuotas are SDK defaults (OpcUaApplicationHost.cs:472), so the oversized response is rejected after the allocation. With MaxConcurrentHistoryReadBatches = 16 this is a trivial memory-exhaustion vector, and it also works on the Gateway backend when the gateway is in streaming mode (the default AllowStreamingBeyondCeiling).
• Suggested fix: Treat 0 as "server-chosen page size" (Part 11 allows a server to return fewer values plus a continuation point): add a ServerHistorian:MaxRawValuesPerRead-style hard cap (advertise it as HistoryServerCapabilities/MaxReturnDataValues) and route N = 0 through the existing RawPageLimit paging. Independently, bound the backends' TOP/LIMIT/max_values by that cap so a misbehaving caller cannot bypass the node manager.

1.2 Gateway events read drops maxEvents on the wire and abandons the gRPC stream at the cap

• Severity: High Confidence: Medium (the eviction behaviour is asserted by the repo's own comments and the gateway's HistorianRpcFaults, not re-verified here)
• Location: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/HistorianGatewayClientAdapter.cs:111-124; GatewayHistorianDataSource.cs:576-598; contrast RawAggregateReader.cs:18-24 and GatewayHistorianDataSource.cs:202-204.
• Evidence:

  // Adapter: maxEvents is accepted and never forwarded (the 0.3.0 package API has no max parameter)
  public IAsyncEnumerable<HistorianEvent> ReadEventsAsync(string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken ct)
  { ... return _inner.ReadEventsAsync(startUtc, endUtc, filter, ct); }
  // Data source: breaks out of the server stream once the cap is full
  if (hasCap && collected.Count == maxEvents) { truncated = true; break; }
  

  while the raw path documents: "Every stream is consumed to completion (never broken out of — a cancelled gateway stream evicts its pooled historian session)" and RawAggregateReader says "the gateway currently classifies that cancellation as a connection fault and evicts its shared historian session (HistorianGateway HistorianRpcFaults.IsConnectionFault: any RpcException ⇒ true)".
• Failure scenario: Every capped HistoryReadEvents that has more than NumValuesPerNode events in its window (the normal case for a consumer paging alarm history, and for the #691.4 reverse walk-back which issues many small capped reads) cancels the server stream mid-way ⇒ the gateway evicts its historian session ⇒ the next read of any kind pays a reconnect, and concurrent raw reads sharing that session fail. Uncapped reads (int.MaxValue) additionally stream the entire window regardless of what the client asked for.
• Suggested fix: Drain the stream to completion with a bounded budget like the raw path (count past the cap without collecting), and add a max_events field to the gateway RPC so the gateway stops streaming at the cap; until then at least set EventReadHorizon guidance in docs.

1.3 FasterLogHistorizationOutbox.PeekBatchAsync holds a monitor across FasterLog disk reads, blocking the recorder actor's appends

• Severity: Low Confidence: High
• Location: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/Recorder/FasterLogHistorizationOutbox.cs:215-252 (peek) and :147-156 (append index under the same lock); drain runs off-actor via DrainOnceAsync(...).PipeTo at src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:696.
• Evidence:

  lock (_state)
  {
      ...
      using FasterLogScanIterator iter = _log.Scan(_nextScanAddress, _log.TailAddress, recover: false);
      while (batch.Count < max && iter.GetNext(out byte[] bytes, out int len, out long address, out long next))
  

• Failure scenario: FasterLog is built with default settings (new FasterLogSettings { LogDevice = _device }, line 76), i.e. an ~8 MB in-memory window; Capacity defaults to 500 000 entries (≈35 MB on disk). Once the backlog's head has aged out of memory (any multi-minute gateway outage), GetNext reads pages from the device synchronously while _state is held; the actor thread's AppendBatchAsync then blocks on lock (_state) for the duration of the read — a sync block on an Akka dispatcher thread, once per 2 s drain pass, exactly during outage recovery.
• Suggested fix: Scan outside the lock (snapshot _nextScanAddress + the live-list cursor under the lock, iterate unlocked, re-take the lock only to flag/drop unreadable head records), or size MemorySizeBits to the configured capacity.

1.4 Client raw history read leaves the server continuation point unreleased when stopped by the client cap

• Severity: Low Confidence: High
• Location: src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs:175-206 vs the modified-read path :263-272.
• Evidence: while (continuationPoint != null && continuationPoint.Length > 0 && allValues.Count < maxValues); then return allValues; — no releaseContinuationPoints call; HistoryReadModifiedAsync explicitly releases ("A read stopped by the client's own cap leaves a server cursor behind; release it.").
• Failure scenario: A page that returns exactly maxValues (the common case on a dense tag) carries a continuation point the client never resumes or releases. Bounded in practice: the server's store is per-session and capped at MaxHistoryContinuationPoints (100, oldest-evicted, OpcUaServer/HistoryContinuationStore.cs:50-55), so the Client.UI's long-lived session just cycles 100 slots; the CLI disconnects immediately.
• Suggested fix: Mirror the modified path's release call.

1.5 Client subscription adapters delete server-side but never remove the Subscription from the SDK Session

• Severity: Low Confidence: Medium
• Location: src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs:321-333 (_session.AddSubscription(subscription)), DefaultSubscriptionAdapter.cs:104-117, 122-133 (_subscription.DeleteAsync(true, ct) only).
• Failure scenario: OpcUaClientService.SubscribeAlarmsAsync/UnsubscribeAlarmsAsync and MainWindowViewModel.MonitorAlarmsForSelectedNodeAsync create a fresh SDK Subscription per cycle; each is deleted on the server but stays in Session.Subscriptions until the session is disposed, so a UI session that re-targets alarms repeatedly accumulates dead Subscription objects (and anything the SDK does over Session.Subscriptions, e.g. transfer/recreate on its own reconnect, iterates them).
• Suggested fix: Call _session.RemoveSubscription(subscription) after a successful delete (the adapter would need the session reference), or create subscriptions through Session.RemoveSubscription-aware helpers.

---

2. Unimplemented / stubbed / silently wrong

2.1 Gateway native TimeAverage / Total are served with Good status although the repo documents the values are wrong until historiangw#27

• Severity: Medium Confidence: High
• Location: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/NativeAggregatePlan.cs:39-43; served via GatewayHistorianDataSource.cs:338-343, 350-387; advertised in ServedAggregates (:40-43).
• Evidence:

  ⚠️ Today historiangw#27 (the interval reaches AVEVA in ticks where it reads milliseconds) makes AVEVA answer ONE
  Good row per request — the whole-intervals read lands on its earliest interval with a whole-span value and the rest
  are absent — in both directions.
  

  (.claude/rules/historian.md agrees: "until historiangw#27 lands both directions return Good-status wrong values".)
• Failure scenario: A client asks for 24 hourly TimeAverage intervals and receives one Good, whole-day value stamped on the first interval and 23 absent intervals; nothing on the wire says the answer is degraded. ScadaBridge or a report will store it as truth.
• Suggested fix: Until the gateway fix ships, either drop TimeAverage/Total from ServedAggregates (⇒ BadAggregateNotSupported, consistent with the other five time-weighted aggregates on this backend) or mark every row UncertainDataSubNormal + log once. Gate the native path behind a config flag that defaults off until #27 is live-verified.

2.2 Gateway backend capability gaps vs. SqlServer/QuestDb (documented, but a consumer-visible matrix)

• Severity: Medium Confidence: High
• Location: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianProvider.cs:54 (CreateUpdater => null), GatewayHistorianDataSource.cs:40-43 (12 of 17 aggregates), :125-126 and :556-562 (reverse raw/events NotSupportedException), ReadModifiedAsync not overridden (interface default ⇒ unsupported), no CreateRetention, GatewayHistorianValueWriter.cs:28-38 (quality discarded by the gateway — "C-002"), Mapping/HistorianTypeMapper.cs (String/DateTime not provisioned).
• What a client sees on Gateway only: HistoryUpdate ⇒ BadHistoryOperationUnsupported; IsReadModified ⇒ unsupported; TimeAverage2/Total2/Interpolative/PercentGood/PercentBad ⇒ BadAggregateNotSupported; every stored sample reads back Good (comms-loss markers are dropped, driver Bad statuses are lost); reverse/end-only raw and events reads are emulated by forward walks (ReverseRawReader/ReverseEventsReader) that cost many round trips on sparse tags.
• Suggested fix: Nothing to "implement" in this repo for most of it (gateway-side), but publish the matrix in docs/Historian.md and have the AdminUI backend picker show it; the quality loss in particular deserves a startup Warning when ContinuousHistorization is enabled on Gateway.

2.3 Client CLI/UI offer StandardDeviation, which no backend serves, and omit six aggregates that are served

• Severity: Low Confidence: High
• Location: src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs (12 members incl. StandardDeviation); Client.CLI/Commands/HistoryReadCommand.cs:50, 202-221; Client.UI/ViewModels/HistoryViewModel.cs:47-62. Served set: Sql.Common/AggregateFamily.cs:50-51 (FamilyOf != null ⇒ 17: Average/Min/Max/Count/Range/Start/End/Delta/Min-MaxActualTime/TimeAverage/TimeAverage2/Total/Total2/Interpolative/PercentGood/PercentBad).
• Failure scenario: historyread --aggregate stddev always returns BadAggregateNotSupported; Range, Delta, MinimumActualTime, MaximumActualTime, PercentGood, PercentBad cannot be requested from the project's own client at all, so the gate recipes in docs/Historian.md cannot be run with Client.CLI.
• Suggested fix: Align AggregateType + AggregateTypeMapper with AggregateFamilies.Served.

2.4 ProbeAsync / GetConnectionStatusAsync exist on the gateway seam but health is derived from read outcomes only

• Severity: Low Confidence: High
• Location: IHistorianGatewayClient.cs:106, 111; GatewayHistorianDataSource.cs:616-638 (ProcessConnectionOpen: _totalSuccesses > 0 && _consecutiveFailures == 0).
• Failure scenario: A freshly booted node with a healthy gateway reports the historian as closed until some client performs a HistoryRead; a node that only writes (continuous historization) never flips it. Operators reading /hosts get a false negative.
• Suggested fix: Use ProbeAsync on a slow timer (or on GetHealthSnapshot with a cached result) now that the seam carries it.

2.5 Two Gateway read arms record a caller cancellation as a backend failure

• Severity: Low Confidence: High
• Location: GatewayHistorianDataSource.cs:607-612 (ReadEventsAsync) and :519-524 (ReadAtTimeNativeAsync) lack the catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) throw; that ReadRawAsync/ReadProcessedAsync/ReadAtTimeAsync have (:141-144, 377-380, 484-487).
• Failure scenario: A HistoryReadDeadline expiry or client abort during an events read bumps TotalFailures/ConsecutiveFailures and flips ProcessConnectionOpen to false ⇒ health alarms on a healthy gateway.

---

3. Performance limiters

3.1 QuestDB live-write path: three round trips plus a 10 ms-poll visibility wait per tag group per drain pass

• Severity: Medium Confidence: High
• Location: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianValueWriter.cs:91-102, 128-143; QuestDbWalVisibility.cs:49-91.
• Evidence:

  await _catalog.EnsureAsync(tag, dataType, ct);            // cached after first
  var present = await ExistingTicksAsync(tag, rows, ct);    // SELECT ts ... per cluster
  for (...) await InsertAsync(...);                         // multi-row INSERT
  await _wal.WaitAsync(_conn.SampleTable, ct);              // SELECT sequencerTxn, then poll writerTxn every 10 ms ≤ 5 s
  

• Failure scenario: With DrainConcurrency = 8 and WAL apply lagging under load, up to eight pollers each issue ~100 wal_tables() queries/s (≈800/s) against the same database that is already behind — the wait amplifies load exactly when the DB is slowest, and every tag group pays ≥3 round trips, so a 5 000-tag deployment at 1 s cadence spends ⌈5000/8⌉ × (3 RTT + apply latency) per pass and falls behind the 2 s cadence. The existence pre-read is only needed for first-wins idempotency across redeliveries, and the visibility wait only so that an immediate redelivery sees the rows.
• Suggested fix: Wait once per drain pass (expose a FlushAsync/pass-end hook on IHistorianValueWriter, or let the recorder call WaitAsync once after Task.WhenAll), back the poll off exponentially (10→50→200 ms), and consider a per-pass existence read keyed by the outbox batch rather than per tag.

3.2 All three alarm-history writers insert one event per round trip, serially

• Severity: Medium Confidence: High
• Location: Gateway/GatewayAlarmHistorianWriter.cs:66-78 (one SendEventAsync per event); SqlServer/SqlServerAlarmHistorianWriter.cs:55-59, 76-89 (new pooled connection + single-row INSERT … WHERE NOT EXISTS per event); QuestDb/QuestDbAlarmHistorianWriter.cs:64-69, 89-105.
• Failure scenario: The LocalDbStoreAndForwardSink drains BatchSize events per tick; an alarm storm of 10 000 events drains at 10 000 × RTT (≥ 10–20 s on SQL, minutes over a WAN to the gateway), and the Primary-only drain means the backlog sits in LocalDb meanwhile. The per-event design exists to isolate poison events, but SqlServer already has the TVP + OUTPUT machinery to get per-row outcomes from one statement.
• Suggested fix: SqlServer: one INSERT … SELECT FROM @events WHERE NOT EXISTS … with OUTPUT inserted.EventId and map outcomes by id (fall back to per-row only on a data error). QuestDb: multi-row INSERT (DEDUP key already converges duplicates). Gateway: needs a batch SendEvents RPC.

3.3 SQL Server retention issues O(tags) DELETE TOP statements per purge even when nothing is eligible

• Severity: Low Confidence: High
• Location: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs:89-107, 134-149.
• Evidence: SELECT TagId FROM Tag then foreach (var id in tagIds) { DeleteLoopAsync(Sample…); if (trail) DeleteLoopAsync(SampleModified…); } — the loop always runs at least one statement per table per tag, and the applock is held for the whole pass.
• Failure scenario: 50 000 historized tags ⇒ ≥100 000 statements per hourly purge (plus the full Event scan) on a database that has nothing to delete, serialised on one connection while other nodes' purges skip.
• Suggested fix: Pre-filter: SELECT TagId FROM Tag t WHERE EXISTS (SELECT 1 FROM Sample s WHERE s.TagId = t.TagId AND s.SourceTimeUtc < @cutoff) (a key seek per tag inside one statement), or keep a per-tag OldestSampleUtc high-water mark.

3.4 QuestDB event.source and sample_modified.tag symbols carry no index

• Severity: Low Confidence: High
• Location: QuestDb/QuestDbTables.cs:73-77 (source SYMBOL), :95-99 (tag SYMBOL), vs the sample table's tag SYMBOL INDEX added for #683 (:64-67, :116).
• Failure scenario: ReadEventsAsync with a sourceName filter (QuestDbHistorianDataSource.cs:402-407) and every ReadModifiedAsync (:183-185) scan all partitions in the window rather than seeking; the #683 gate measured the same shape costing seconds on the sample table.
• Suggested fix: Add INDEX on both symbols in the DDL plus the same idempotent ALTER … ADD INDEX migration the bootstrapper already runs for sample.tag.

3.5 GatewayTagPresenceCache adds a catalog-browse RPC before HistoryReads, keyed by a glob filter

• Severity: Low Confidence: Medium
• Location: Gateway/GatewayTagPresenceCache.cs:16-32 (60 s positive / 5 s negative TTL, 4 096 entries); HistorianGatewayClientAdapter.cs:85-93 (BrowseTagNamesAsync(tag, ct) drained fully). Package doc (0.3.0 XML): BrowseTagNamesAsync(filter) — "Tag name filter pattern accepted by the historian (e.g. Sys*)".
• Failure scenario: A consumer sweeping history for N tags pays N extra streaming RPCs per minute of sweep; with more than 4 096 distinct tags the LRU-less eviction (_entries.Keys.First()) thrashes. A historian tag name containing */? (legal in AVEVA) streams every matching catalog entry on each check.
• Suggested fix: Use the package's GetTagMetadataAsync (exact lookup) now that the seam exists, lengthen the positive TTL, or drop the pre-check and treat an empty raw read as "no data" (which it already does).

3.6 Client: N+1 browse in BrowseAsync, and O(n) scan per notification in the UI

• Severity: Low Confidence: High
• Location: Client.Shared/OpcUaClientService.cs:223-242 (HasChildrenAsync ⇒ one Browse per Object child); Client.UI/ViewModels/SubscriptionsViewModel.cs:61-73 (foreach (var item in ActiveSubscriptions) if (item.NodeId == e.NodeId) on the UI thread per notification).
• Failure scenario: subscribe --recursive / the UI's recursive subscribe over a 50 000-node raw tree issues ~50 000 Browse calls to find children; 2 000 subscribed items at 1 Hz = 4 M string compares/s on the Avalonia UI thread.
• Suggested fix: Browse with includeSubtypes/resultMask carrying NodeClass only and infer HasChildren lazily on expand; index ActiveSubscriptions by NodeId in a dictionary.

---

4. Other hazards

4.1 Every production IHistorianValueWriter swallows OperationCanceledException, so the recorder's shutdown guard is dead code

• Severity: Medium Confidence: High
• Location: Gateway/Recorder/GatewayHistorianValueWriter.cs:112-120; SqlServer/SqlServerHistorianValueWriter.cs:93-97; QuestDb/QuestDbHistorianValueWriter.cs:104-108 — all catch (Exception) { …; return false; }. The consumer expects the opposite: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ContinuousHistorizationRecorder.cs:744-755:

  catch (OperationCanceledException) when (ct.IsCancellationRequested)
  {
      // Shutdown cut this write short. ... swallowed as `ok = false` it becomes a consecutive failure, and when a
      // sibling group already wrote this pass the fold books it — so a routine node stop can tip a perfectly
      // HEALTHY tag over its attempt budget and discard its queued samples on the way out. Rethrow ...
      throw;
  }
  

• Failure scenario: At shutdown mid-pass the cancelled groups return false; if any sibling already wrote, _tagWriteFailures[tag]++ for each cancelled tag (recorder :826-834) — precisely the outcome the guard was written to prevent. Today the damage is bounded (+1 per process stop, counters are in-memory, _lifetime cancels only in PostStop, recorder :340), so no data is lost, but the contract between writer ("never throws") and recorder ("rethrow cancellation") is contradictory and the next refactor that cancels drains without restarting (a capture-gate close, a backend switch) will discard healthy tags' backlog after MaxWriteAttempts such events.
• Suggested fix: catch (OperationCanceledException) when (ct.IsCancellationRequested) { throw; } ahead of the catch-all in all three writers (the alarm writers already special-case it), and state it on IHistorianValueWriter.

4.2 Gateway value writer relabels a Local-kind timestamp as UTC instead of converting it

• Severity: Medium Confidence: Medium (depends on a driver ever producing DateTimeKind.Local)
• Location: Gateway/Recorder/GatewayHistorianValueWriter.cs:93-98:

  live.Timestamp = Timestamp.FromDateTime(DateTime.SpecifyKind(timestampUtc, DateTimeKind.Utc));
  

  vs the SQL backends' shared normaliser Sql.Common/LiveWritePreparer.cs:31: t = t.Kind == DateTimeKind.Local ? t.ToUniversalTime() : DateTime.SpecifyKind(t, DateTimeKind.Utc);.
• Failure scenario: A driver that stamps DateTime.Now-derived source times (or an OpcUaClient upstream server returning Local) historizes correctly on SqlServer/QuestDb but shifted by the node's UTC offset on Gateway — and the HistoryUpdate path on SQL backends keeps the kind (HistoryUpdatePlanner ⇒ LiveWritePreparer.Truncate only), so the same value can land at two different times on two backends.
• Suggested fix: Run gateway writes through LiveWritePreparer.Prepare (it already does the non-finite ⇒ Empty and dedup work the gateway path lacks), or at minimum ToUniversalTime() for Local.

4.3 SqlServer DeleteRaw removes an unbounded range in a single statement/transaction with OUTPUT INTO

• Severity: Low Confidence: High
• Location: SqlServer/SqlServerHistorianDataUpdater.cs:99-113.
• Failure scenario: DELETE FROM Sample OUTPUT … INTO SampleModified WHERE TagId=@id AND SourceTimeUtc >= @s AND < @e over years of a 1 Hz tag deletes tens of millions of rows and writes as many audit rows in one transaction: lock escalation on Sample (blocking the live writer), transaction-log growth, and a 30 s CommandTimeout that aborts and rolls back the whole thing — the retention service deliberately batches DELETE TOP (n) for this reason. QuestDB's updater already pages (QuestDbHistorianDataUpdater.cs:135-166).
• Suggested fix: Page the delete (DELETE TOP (@n) … OUTPUT … in a loop under one transaction or per-batch commits; the OPC UA result is a single status either way).

4.4 QuestDB HistoryUpdate: non-atomic trail + per-instance locks + duplicating retries (documented residuals)

• Severity: Low Confidence: High
• Location: QuestDb/QuestDbHistorianDataUpdater.cs:18-21, 28-34, 155-160, 235-240; .claude/rules/historian.md ("a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key)").
• Failure scenario: A crash or failed sample INSERT after WriteTrailAsync leaves orphan sample_modified rows describing changes that never landed; a second node (or the live writer) racing the same key bypasses the 64 striped SemaphoreSlims; a Npgsql timeout retry duplicates trail rows. All documented; listed so the matrix is complete.
• Suggested fix: Record a applied marker (a second append with the same modified_id) after the sample write so ReadModifiedAsync can drop orphans; use a DEDUP key (ts, tag, modified_id) on the trail.

4.5 Client: failover nulls subscription state outside the lock the rest of the class relies on

• Severity: Low Confidence: High
• Location: Client.Shared/OpcUaClientService.cs:734-735 (_dataSubscription = null; _alarmSubscription = null; in RunFailoverAsync, no lock) vs :260-263 (SubscribeAsync reads/creates _dataSubscription unguarded) and the class comment :18-22 that promises all bookkeeping is under _subscriptionLock.
• Failure scenario: A SubscribeAsync racing a keep-alive failover can add a monitored item to the dying session's subscription (silently lost, not replayed) or observe a null and create a second subscription on the new session that ReplaySubscriptionsAsync then duplicates.
• Suggested fix: Take _subscriptionLock around the null-outs and guard SubscribeAsync with the same snapshot pattern DisconnectAsync uses; also ConnectAsync (:97-128) should refuse (or disconnect) when already connected instead of overwriting _session.

---

5. Checked and fine (looked like problems, are not)

• SQL injection: every dynamic identifier is regex-validated before splicing — SqlServer Schema ^[A-Za-z_][A-Za-z0-9_]{0,127}$ (SqlServerHistorianOptions.cs:49-59, SqlHistorianConnection.Q brackets it, $(Schema) substituted only after validation), QuestDb TablePrefix ^[A-Za-z0-9_]+$ (QuestDbHistorianOptions.cs:32-38, re-checked in QuestDbTables.Name), table_columns('{table}') uses that validated name, QuestDB AtTimeSql inlines only generated ints, TTL days is int. All values (tag names, times, principals, event fields) are parameters; the SqlServer alarm writer deliberately uses NVARCHAR(MAX) params so truncation is a server error, not a silent clip.
• Continuous-historization buffer bound: the FasterLog outbox is disk-backed with Capacity (500 000 default) drop-oldest and DroppedCount/UnreadableCount metrics; drop-oldest truncates + commits, corrupt head records cannot pin the head (FasterLogHistorizationOutbox.cs:172-200, 243-250). Not unbounded.
• Flush on shutdown: PerEntry commit mode fsyncs before AppendBatchAsync returns; Dispose cancels the periodic loop, awaits it, then Commit(spinWait: true) (:365-406). Appends in flight at process kill are the documented Periodic trade-off only.
• Time-weighted aggregates stream, they do not materialise: both SQL backends feed TimeWeightedAggregator row by row with TOP/LIMIT (cap + 1) and CheckWindowRows fails the node one row past MaxTimeWeightedRawRows; memory is O(buckets) (SqlServerHistorianDataSource.cs:528-550, QuestDbHistorianDataSource.cs:545-579, TimeWeightedAggregator.cs:132-154). Gateway per-sample aggregates likewise accumulate into RawBucketAccumulator (O(buckets)).
• BucketPlan.MaxBuckets = 100 000 bounds every processed read's result list and the aggregator arrays before any I/O.
• Tag catalogs (SqlServerTagCatalog._cache, QuestDbTagCatalog._known) are positive-only and never pruned, but are bounded by the number of distinct historian tag names ever written on this node (≤ a few hundred KB); a renamed tag leaves one stale int entry. Not a leak worth fixing.
• Gateway channels: four HistorianGatewayClientAdapters (data source, alarm writer, provisioner, value writer) each own a GrpcChannel, each disposed by its owner (DisposeAsync on all four). Documented as deliberate; a co-located sidecar tolerates four HTTP/2 connections.
• GatewayRowCeiling is a single int + ticks with Volatile/Interlocked; a race costs one extra refusal, never a wrong answer. GatewayAtTimeReader is capped at 64 + 16 × times reads and 1 024 rows per read, so a pathological tag fails BadTooManyOperations rather than looping.
• Schema bootstrap: SqlServerSchemaBootstrapper/QuestDbSchemaBootstrapper use SemaphoreSlim(1,1) + a 5 s negative cache (BootstrapFailureCache), so an outage does not serialise every call behind one round trip; the SqlServer applock is session-owned and the pool is cleared when release fails (otherwise the lock would ride a pooled connection forever) — same pattern in retention.
• Timestamps on the SQL backends: datetime2(7) with explicit UTC SpecifyKind on read/write; QuestDB rounds to µs and nudges Min/MaxValue so Npgsql never binds ±infinity; pre-1970 is refused/dropped with counters rather than failing a batch. IHistorianDataSource.TimestampResolutionTicks = 10 lets the paging step correctly on QuestDB.
• GatewayHistorianDataSource.Dispose() is sync-over-async (DisposeAsync().AsTask().GetAwaiter().GetResult()) — shutdown-only, channel shutdown is quick; acceptable.
• Analyzer (UnwrappedCapabilityCallAnalyzer): registers nothing when the guarded interfaces are absent from the compilation, resolves symbols once per compilation, and walks ancestors only up to the enclosing wrapper invocation; the one acknowledged limitation (any lambda argument to ExecuteAsync suppresses) is documented and currently unreachable. No production hazard.
• Client DisposeBridge bounds every sync-over-async teardown (CloseAsync/DeleteAsync) at 5 s and cancels the token when the wait loses, so a dead server cannot hang Dispose.
• HistorizationOutboxEntrySerializer is bounds-checked for both layouts (negative length prefixes, short records, invalid DateTime binary, non-0/1 booleans all return false), so a corrupt record is counted and skipped, never thrown.